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ABSTRACT 

The  need  for  efficient  storage  and  processing  of  very 
large  databases  to  support  decision-making  coupled  with 
advances  in  computer  hardware  and  software  technology  have 
made  research  and  development  of  specialized  architectures 
for  database  management  a  very  attractive  and  important 
area. 

The  INFOPLEX  data  base  computer  proposed  by  Madnick 
applies  the  theory  of  hierarchical  decomposition  to  obtain  a 
specialized  architecture  for  database  management  with 
substantial  improvements  in  performance  and  reliability  over 
conventional  architectures.  The  storage  subsystem  of 
INFOPLEX  is  realized  using  a  data  storage  hierarchy.  A  data 
storage  hierarchy  is  a  storage  subsystem  designed 
specifically  for  managing  the  storage  and  retrieval  of  very 
large  databases  using  storage  devices  with  different 
cost/performance  characteristics  arranged  in  a  hierarchy. 

It  makes  use  of  locality  of  data  references  to  realize  a  low 
cost  storage  subsystem  with  very  large  capacity  and  small 
access  time. 

As  part  of  the  INFOPLEX  research  effort,  this  thesis  is 
focused  on  the  study  of  high  performance,  highly  reliable 
data  storage  hierarchy  systems.  Concepts  of  the  INFOPLEX 
data  base  computer  are  refined  and  new  concepts  of  data 
storage  hierarchy  systems  are  developed.  A  preliminary 
design  of  a  general  structure  for  the  INFOPLEX  data  storage 
hierarchy  system  is  proposed. 

Theories  of  data  storage  hierarchy  systems  are  developed. 
Madnick's  model  of  a  generalized  storage  hierarchy  is 
extended  and  formalized  for  data  storage  hierarchy  systems. 
The  Least  Recently  Used  (LRU)  algorithm  is  extended  to 
incorporate  the  read-through  strategy  and  page  overflow 
strategies  to  obtain  four  classes  of  data  movement 
algorithms.  These  algorithms  are  formally  defined. 

Important  performance  and  reliability  properties  of  data 
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storage  hierarchy  systems  that  make  use  of  these  algorithms 
are  identified  and  analyzed  in  detail.  It  is  proved  in 
Theorems  1  and  2  that  depending  on  the  relative  sizes  of  the 
storage  levels  and  the  algorithms  used,  it  is  not  always 
possible  to  guarantee  that  the  contents  of  a  given  storage 
level  'i'  is  a  superset  of  the  contents  of  its  immediate 
higher  storage  level  'i-1',  i.e.,  multi-level  inclusion 
(MLI)  does  not  hold.  Necessary  and  sufficient  conditions 
for  MLI  to  hold  are  identified  and  proven  in  Theorems  3  and 
4.  A  property  related  to  MLI  is  the  multi-level  overflow 
inclusion  (MLOI)  property.  MLOI  holds  if  an  overflow  page 
from  storage  level  'i'  is  always  found  to  already  exist  in 
storage  level  ' i+1 ' .  A  data  storage  hierarchy  avoids 
cascaded  references  to  lower  storage  levels  if  MLOI  holds. 
Necessary  and  sufficient  conditions  for  the  MLOI  to  hold  are 
identified  and  proven  in  Theorems  5  and  6.  It  is  possible 
that  increasing  the  sizes  of  intermediate  storage  levels  may 
actually  increase  the  number  of  references  to  lower  storage 
levels,  resulting  in  decreased  performance.  This  is 
referred  to  as  the  multi-level  paging  anomaly  (MLPA) . 
Conditions  necessary  to  avoid  MLPA  are  identified  and  proven 
in  Theorems  7  and  8. 

A  simplified  structure  of  the  INFOPLEX  data  storage 
hiei:>„rchy  is  .drived  from  its  general  structure.  Protocols 
for  supporting  ti.e  read-through  and  store-behind  algorithms 
are  specified.  Two  simulation  models  of  this  system  are 
developed.  The  first  *1-  ".I  incorporates  one  functional 
processor  and  three  storage  levels.  Results  from  this  model 
provide  significant  insights  to  the  design  and  its 
algorithms  and  reveals  a  potential  deadlock  in  the  buffer 
management  schemes.  The  second  model  corrects  this 
potential  deadlock  and  also  incroproates  five  functional 
processors  and  four  storage  levels.  Results  from  this  model 
show  that  the  store-behind  operation  may  be  a  significant 
system  bottleneck  because  of  the  multi-level  inclusion 
requirement  of  the  data  storage  hierarchy.  By  using  more 
parallelism  in  the  lower  storage  levels  and  by  using  smaller 
block  sizes  it  is  possible  to  obtain  a  well-balanced  system 
which  is  capable  of  supporting  the  storage  references 
generated  by  the  INFOPLEX  functional  hierarchy.  The  effects 
of  using  projected  1985  technology  for  the  data  storage 
hierarchy  are  also  assessed. 


Thesis  Super'-.' -'^r :  Prof.  Stuart  E.  Madnick 

Associate  Professor  of  Management  Science 
M.I.T.  Sloan  School  of  Management 
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Chapter  I 

INTRODUCTION  AND  PLAN  OF  THESIS 

1.1  INTRODUCTION 

The  effective  and  efficient  storage  and  processing  of 
very  large  data  bases  to  support  better  decision-making  has 
been  a  major  concern  of  modern  organizations.  Though 
advances  in  computer  technology  are  impressive,  the  rate  of 
growth  of  information  processing  in  organizations  is 
increasing  even  more  rapidly.  A  key  technological  advance 
in  providing  better  information  processing  systems  is  the 
development  of  Data  Base  Management  Systems  (DBMS's)  (Mar¬ 
tin,  1975).  Most  organizations  today  make  use  of  some  kind 
of  DBMS  for  handling  their  large  databases.  Efforts  to 
develop  even  more  effective  DBMS's  remain  very  active  and 
important  (Mohan,  1978). 

Current  DBMS's  are  capable  of  handling  large  databases  on 
the  order  of  trillion  bits  of  data  (Simonson  and  Alsbrooks, 
1975) ,  and  are  capable  of  handling  query  rates  of  up  to  one 
hundred  queries  per  second  (Abe,  1977).  Due  to  the  increas¬ 
ing  need  for  better  information,  and  the  declining  costs  of 
processors  and  storage  devices,  it  is  expected  that  future 
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high  performance  DBMS's  will  be  required  to  handle  query 
rates  and  provide  storage  capacities  several  orders  of  mag¬ 
nitude  higher  than  today's  (Madnick,  1977).  Furthermore, 
with  such  high  query  rates  (generated  by  terminal  users  as 
well  as  directly  from  other  computers) ,  it  is  essential  that 
a  DBMS  maintains  non-stop  operation  (Computerwor Id ,  1976). 
Thus,  guaranteeing  the  reliability  of  the  DBMS  becomes  very 
difficult. 

Current  improvements  in  processor  and  storage  device 
technology  alone  do  not  seem  to  be  able  to  meet  these  orders 
of  magnitude  improvements  in  performance  and  reliability. 

The  nex  section  reviews  several  research  efforts  aimed  at 
modifying  the  conventional  computer  architecture  for  better 
information  handling.  One  such  research  effort  is  the  INFO- 
PLEX  Project  (Lam  and  Madnick,  1979).  The  INFOPLEX  approach 
to  obtaining  a  high  performance,  highly  reliable  DBMS  is  to 
design  a  new  computer  specifically  for  data  management. 

This  thesis  is  a  study  of  the  storage  subsystem  of  the  INFO¬ 
PLEX  data  base  computer.  Research  goals  and  specific  accom¬ 
plishments  of  this  thesis  will  be  described  in  a  following 
section.  The  structure  of  this  thesis  is  then  described. 
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1.2  RELATED  RESEARCH 


In  the  past,  computers  were  designed  primarily  for  numer¬ 
ical  computation.  We  now  find  that  processing  of  large 
databases  has  become  a  major,  if  not  dominant,  component  of 
computer  usage.  However,  current  computer  structures  still 
have  the  'von  Neumann'  structure  of  twenty  years  ago.  As 
Mueller  (Mueller,  1976),  President  of  System  Development 
Corporation,  noted:  'the  computer  industry  has  gone  through 
three  generations  of  development  to  perfect  machines  optim¬ 
ized  for  10  percent  of  the  workload'.  It  is  not  surprising 
then,  that  many  organizations  find  their  supercomputers  run¬ 
ning  'out  of  steam'  as  new  applications  with  large  databases 
are  installed. 

Figure  1.1  illustrates  a  simplified  typical  computer 
architecture.  It  consists  of  a  processor  directly  accessing 
a  main  memory  (with  access  time  in  the  order  of  microse¬ 
conds)  ,  an  I/O  processor  that  controls  the  movement  of  data 
between  main  memory  and  secondary  storage  devices,  an  I/O 
controller  and  its  associated  secondary  storage  devices 
(with  access  times  in  the  order  of  milliseconds) .  Current 
DBMS's  are  software  systems  that  reside  in  the  main  memory 
together  with  other  software  subsystems  and  application  pro¬ 
grams.  To  provide  a  high  level  view  of  data  for  application 
programs  a  DBMS  has  to  manage  all  the  data  residing  in  sec- 
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Figure  1.1  A  Typi 


1  Computer  Architecture 


ondary  storage  devices  and  coordinate  all  the  data  movement 
and  processing  activities.  Two  potential  deficiencies  of 
adapting  the  conventional  computer  architecture  for  data 
base  management  become  evident.  First,  the  processor 
becomes  strained  as  new  functions  are  added  to  the  DBMS. 
These  new  functions  include  high  level  language  support, 
better  security  and  data  integrity  mechanisms,  support  of 
multiple  data  models,  ...,  and  so  on.  Second ,  due  to  the 
large  differential  in  access  times  of  main  memory  and  secon¬ 
dary  storage  devices  (referred  to  as  the  'access  gap'),  the 
speed  of  processing  becomes  limited  by  how  fast  'useful  data 
can  be  brought  into  main  memory  from  secondary  storage  dev¬ 
ices.  Thus,  many  organizations  find  the  performance  of 
their  data  management  system  either  limited  by  the  available 
processor  cycles  or  limited  by  the  speed  of  I/O  operations, 
depending  on  the  DBMS  used  and  the  applications  supported. 

These  problems  have  been  recognized  for  some  time.  Cur¬ 
rent  advances  in  LSI  technology  make  it  feasible  to  consider 
new  software  and  hardware  architectures  to  overcome  the 
above  deficiencies.  Several  such  approaches  are  discussed 
below. 
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1.2.1  New  Instructions  Through  Microprogramming 

Conventional  processor  instructions  are  usually  not  well 
suited  to  the  requirements  of  database  management  systems. 
Using  firmware,  it  is  possible  to  augment  or  enhance  the 
instructions  thus  effectively  increase  the  efficiency  of  the 
processor.  This  approach  has  been  adopted  in  several  sys¬ 
tems.  One  Of  the  earliest  efforts  occurred  as  part  of  the 
LISTAR  information  retrieval  system  developed  at  M.I.T.'s 
Lincoln  Laboratory  (Armenti  £t  £l . ,  1970),  where  several 
frequently  used  operations,  such  as  a  generalized  List 
Search  operation,  were  incorporated  into  the  microcode  of  an 
IBM  System/360  Model  67  computer.  The  Honeywell  H60/64  uses 
special  instructions  to  perform  data  format  conversion  and 
hashing  corresponding  to  frequently  used  subroutines  of 
Honeywell's  IDS  database  system  (Bachman,  1975).  More 
recently  the  IBM  System/38  (Soltis  and  Hoffman,  1979)  was 
announced  with  microcode  to  perform  much  of  the  operating 
system  and  data  management  functions.  The  performance 
advantages  of  this  approach  are  highly  dependent  upon  the 
frequency  of  use  of  the  new  instructions  and  the  extent  to 
which  they  fit  into  the  design  of  the  overall  database  sys¬ 
tem  software. 
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1.2.2  Storage  Hierarchy  Optimization 

It  is  possible  to  close  the  'access  gap'  between  main 
memory  and  secondary  storage  devices  by  using  a  more  conti¬ 
nuous  storage  hierarchy,  thus  improving  the  performance  of 
the  storage  subsystem. 

Madnick  (Madnick,  1973)  proposed  a  model  of  a  generalized 
storage  hierarchy  system  and  its  data  movement  algorithms. 
This  storage  hierarchy  makes  use  of  multiple  page  sizes 
across  the  storage  levels  for  high  performance  and  maintains 
multiple  data  redundancy  across  the  storage  levels  for  high 
performance  and  high  reliability.  This  type  of  storage 
hierarchy  systems  have  great  potentials  as  storage  subsys¬ 
tems  for  high  performance,  highly  reliable  DBMS's.  Unfortu¬ 
nately,  the  lack  of  better  understanding  of  this  type  of 
storage  hierarchy  systems  is  a  major  obstacle  in  the  devel¬ 
opment  of  practical  storage  subsystems  in  spite  of  the  fact 
that  a  continuous  spectrum  of  storage  devices  with  different 
cost/performance  characteristics  will  persist  (Dennis  et . 
al . ,  1978;  Hoagland,  1979;  Smith,  1978a). 

There  has  been  much  work  on  studying  storage  hierarchy 
systems  and  their  algorithms.  We  shall  review  these  work  in 
a  later  chapter.  These  studies  usually  do  not  consider  the 
effects  of  multiple  page  sizes  across  different  storage  lev¬ 
els,  nor  the  problems  of  providing  multiple  data  redundancy 
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across  the  storage  levels,  as  in  the  system  proposed  by 
Madnick.  Developing  theories  for  generalized  storage  hier¬ 
archy  systems  specifically  for  managing  large  database 
remains  a  challenge. 

1.2.3  Intelligent  Controllers 

Another  approach  to  improving  information  processing 
efficiency  is  to  use  intelligent  controllers.  The  control¬ 
ler  provides  an  interface  between  the  main  memory  and  the 
devices.  Recently,  more  and  more  intelligence  has  been 
introduced  into  these  controllers.  For  example,  many  con¬ 
trollers  can  perform  the  search  key  operation  themselves 
(Ahern  ejt  ^.  ,  1972;  Lang  ^  f  1977).  Since  only 
selected  data  items  are  brought  to  the  main  memory,  the  I/O 
traffic  is  reduced  and  the  efficiency  of  the  storage  subsys 
tern  is  increased. 

Two  major  types  of  intelligent  controllers  have  emerged. 
The  first  type  specializes  in  automating  the  data  transfer 
between  the  storage  devices,  i.e.,  the  physical  storage  man 
agement  functions.  For  example,  IBM's  3850  Mass  Storage 
System  (Johnson,  1975)  uses  an  intelligent  controller  to 
automatically  transfer  data  between  high-capacity,  slow- 
speed  tape  cartridges  and  medium-capacity,  fast  moving-head 
disks.  Thus,  the  processor  is  relieved  of  the  burden  to 
manage  these  data  movements. 
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The  second  type  of  intelligent  controllers  is  designed  to 
handle  some  of  the  logical  storage  management  functions, 
such  as  searching  for  a  specific  data  record  based  on  a  key. 
This  latter  type  of  device  is  sometimes  referred  to  as  a 
database  computer,  and  is  often  used  to  perform  associative 
or  parallel  searching  (Langdon,  1978) .  Most  parallel  asso¬ 
ciative  search  strategies  are  based  on  a  head-per-track  sto¬ 
rage  device  technology  (for  example,  magnetic  drums,  LSI 
shift  registers,  and  magnetic  bubbles)  and  a  multitude  of 
comparators.  As  each  data  record  rotates,  either  mechani¬ 
cally  or  electronically,  past  a  read/write  head,  it  is  com¬ 
pared  with  a  match  record  register,  called  the  mask .  Exam¬ 
ples  of  this  type  of  intelligent  controllers  include  CASSM 
(Copeland  £t  £l. f  1973;  Healy  ^  f  1972;  Su  and  Lipovski, 
1975;  Su,  1977;  Su  et.  ^. ,  1979),  the  Rotating  Associative 
Relational  Storage  (RARES)  design  (Lin  et  ^. ,  1976),  and 
the  Rotating  Associative  Processor  (RAP)  (Ozkarahan  £t  al. , 
1975;  Schuster  £t  ^. ,  1976;  Ozkarahan  ^  £l • r  1977;  Schus¬ 
ter,  1978;  Schuster,  et.  ^. ,  1979). 

Although  the  decline  in  the  costs  of  comparator  electron¬ 
ics,  due  to  advances  in  LSI  technology,  makes  parallel 
search  strategies  quite  promising  for  the  future,  they  are 
only  well  suited  to  storage  technologies  that  lend  them¬ 
selves  to  low  cost  read/write  mechanisms,  and  for  optimal 
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performance  and  operation  they  tend  to  require  a  fairly  sim¬ 
ple  and  uniform  database  structure  (e.g.,  relational  flat 
files).  To  use  these  intelligent  controllers  in  conjunction 
with  other  storage  devices,  such  as  mass  storage,  some 
"staging"  mechanisms  have  to  be  used.  Furthermore,  these 
intelligent  controllers  only  support  part  of  the  information 
management  functions,  much  of  the  complex  functions  of  lan¬ 
guage  interpretation,  support  of  multiple  user  interfaces, 
etc.,  of  an  information  management  system  cannot  easily  be 
performed  in  these  controllers. 

1.2.4  Back-end  Processors 

The  fourth  approach  is  to  shift  the  entire  database  man¬ 
agement  function  from  the  main  computer  to  a  dedicated  com¬ 
puter  thus  increasing  the  processor  power  available  for  per¬ 
forming  the  data  management  function.  Such  a  computer  is 
often  called  a  back-end  processor.  The  back-end  processor 
is  usually  a  minicomputer  specifically  programmed  to  perform 
all  of  the  functions  of  the  database  management  system. 

Back-end  processors  have  evolved  rapidly  in  recent  years. 
Some  of  the  earliest  experimental  efforts  include  the 
loosely  coupled  DATACOMPUTER  (Marill  and  Stern,  1975), 
developed  by  the  Computer  Corporation  of  America  using  the 
DECSystem-10  computer,  and  the  tightly  coupled  XDMS  (Canady 
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et  al. ,  1974),  developed  by  Bell  Laboratories  by  modifying 
the  firmware  of  a  Digital  Scientific  META-4  minicomputer. 
More  recent  developments  include  the  Cullinane  Corporation's 
IDMS  on  a  PDP/11  compuater.  Since  the  back-end  processor  is 
still  a  conventional  computer  whose  architecture  has  been 
designed  for  computational  purposes,  not  for  information 
management,  its  performance  is  still  quite  limited. 

1.2.5  Data  Base  Computers 

The  fifth  approach  to  providing  improved  information  pro¬ 
cessing  efficiency  is  the  database  computer.  The  difference 
between  this  approach  and  the  fourth  approach  (back-end  pro¬ 
cessor)  is  that  the  database  computer  has  a  system  architec¬ 
ture  particularly  suitable  for  database  operations  while  a 
back-end  processor  merely  adapts  a  conventional  computer  to 
database  applications. 

There  has  been  relatively  little  research  on  the  develop¬ 
ment  of  true  database  computers  (as  opposed  to  work  on 
intelligent  controllers  and/or  dedicated  back-end  processors 
—  which  are  sometimes  referred  to  as  database  computers) . 
Current  data  base  computer  research  efforts  include  the  DBC 
(Hsiao  and  Kannan ,  1976;  Banerjee,  e_t.  ^. ,  1978;  Banerjee, 
et .  al . ,  1979)  at  the  Ohio  State  University,  the  CDS  (Hako- 
zaki  et  al.,  1977)  at  the  Nippon  Electric  Co.,  Japan,  and 
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the  INFOPLEX  effort  at  M.I.T.  (Madnick,  1975b;  Lam  and  Mad- 
nick,  1979;  Madnick,  1979). 

Data  Base  Computer  seems  to  be  a  long  term  solution  to 
the  DBMS  requirements  of  future  computer  systems.  The  DBC 
approach  at  Ohio  State  University  makes  use  of  specialized 
functional  processors  for  performing  the  data  management 
functions  thus  eliminating  the  processor  bottleneck  that 
exists  in  current  DBMS's.  To  improve  the  efficiency  of  the 
storage  subsystem,  the  DBC  makes  use  of  the  idea  of  a  parti¬ 
tioned  content  addressable  memory  (PCAM) .  The  entire 
address  space  is  divided  into  partitions,  each  of  which  is 
content  addressable.  To  realize  content  addressability  cost 
effectively,  the  DBC  makes  use  of  multiple  intelligent  con¬ 
trollers  at  the  secondary  storage  devices. 

The  INFOPLEX  architecture  also  makes  use  of  multiple 
functional  processors.  However,  to  obtain  a  flexible,  high 
performance,  and  highly  reliable  storage  subsystem,  INFOPLEX 
makes  use  of  a  storage  hierarchy  system  based  on  the  Madnick 
proposal  (Madnick,  1973).  Conceptually,  the  INFOPLEX  data¬ 
base  computer  consists  of  a  functional  hierarchy  and  a  phy¬ 
sical  (storage)  hierarchy  (See  Figure  1.2).  The  INFOPLEX 
functional  hierarchy  is  a  hierarchy  of  specialized  micropro¬ 
cessors.  It  implements  all  the  information  management  func¬ 
tions  of  a  database  manager,  such  as  query  language 
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interpretation,  security  verification,  and  data  path 
accessing,  etc.  The  hierarchy  of  functional  processors 
establishes  a  pipeline.  Within  each  stage  of  the  pipeline, 
multiple  processors  are  used  to  realize  parallel  processing 
and  provide  multiple  redundancy.  The  INFOPLEX  storage  hier¬ 
archy  is  designed  specifically  to  support  the  data  storage 
requirements  of  the  functional  hierarchy.  To  provide  high 
performance  and  high  reliability,  it  makes  use  of  a  highly 
parallel  and  reliable  architecture,  implements  distributed 
control  mechanisms,  and  maintains  multiple  data  redundancy. 


1.3  RESEARCH  GOALS  AND  ACCOMPLISHMENTS 

This  thesis  is  a  study  of  the  INFOPLEX  data  storage  hier¬ 
archy.  We  have  studied  data  storage  hierarchy  systems  from 
five  different  and  related  points  of  view:  (1)  development 
of  concepts  for  INFOPLEX  and  data  storage  hierarchy  systems, 

(2)  architectural  design  of  data  storage  hierarchy  systems, 

(3)  theoretic  analysis  of  data  storage  hierarchy  systems, 

(4)  algorithm  development  for  data  storage  hierarchy  sys¬ 
tems,  and  (5)  performance  evaluation  of  data  storage  hier¬ 
archy  systems.  Specific  goals  and  accomplishments  of  this 
thesis  are  listed  below. 
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1.  Develop  and  extend  concepts  of  data  base  computers 
and  data  storage  hierarchy  systems;  Since  Mad- 
nick's  (Madnick, 1975b)  proposal  to  develop  a  high 
performance,  highly  reliable  data  base  computer, 
called  INFOPLEX,  there  has  been  many  alternative 
approaches  to  develop  special  architectures  for 
data  base  management.  We  have  reviewed  these  pro¬ 
posals  and  categorized  these  efforts  into;  (1) 
new  instructions  through  microprogramming,  (2) 
storage  hierarchy  optimization,  (3)  intelligent 
controllers,  (4)  back-end  processor,  and  (5)  data 
base  computers.  Concepts  of  the  INFOPLEX  data 
base  computer  have  been  refined  and  leads  to  the 
development  of  the  concept  of  a  data  storage  hier¬ 
archy. 

2.  Architectural  design  of  data  storage  hierarchy 
systems;  Although  storage  hierarchy  systems  with 
two  or  three  levels  are  very  common  in  current 
computer  systems,  there  is  no  known  storage  hier¬ 
archy  with  more  than  three  storage  levels  that  has 
been  designed  specifically  for  large  databases.  A 
preliminary  design  of  the  general  structure  of  a 
data  storage  hierarchy  with  an  arbitrary  number  of 
storage  levels  has  been  developed.  This  structure 
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is  the  basis  for  future  designs  of  data  storage 
hierarchy  systems  for  the  INFOPLEX  data  base  com¬ 
puter. 

3.  Theoretic  analysis  of  data  storage  hierarchy  sys¬ 
tems:  Madnick  (Madniclr^^  1973)  proposed  the  model 
of  a  generalized  storage  hierarchy  system  that 
incorporates  multiple  page  sizes  and  maintains 
multiple  data  redundancy  for  high  performance  and 
high  reliability.  This  model  is  extended  and  for¬ 
malized  for  data  storage  hierarchy  systems.  The 
Least  Recently  Used  (LRU)  algorithm  is  extended  to 
incorporate  the  read-through  strategy  for  managing 
the  data  movement  in  data  storage  hierarchy  sys¬ 
tems.  Four  classes  of  algorithms  are  obtained  and 
formally  defined.  The  multi-level  inclusion 
(MLI),  multi-level  overflow  inclusion  (MLOI),  and 
multi-level  paging  anomaly  (MLPA)  properties  of 
data  storage  hierarchy  systems  using  these  algor¬ 
ithms  are  analyzed  in  detail  and  formally  proved 
as  eight  theorems  and  nine  lemmas. 

4.  Develop  algorithms  for  data  storage  hierarchy  sys¬ 
tems;  A  simplified  structure  of  the  INFOPLEX  data 
storage  hierarchy  is  obtained  from  the  general 
structure.  Protocols  to  support  the  read-through 
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and  the  store-behind  data  movement  algorithms  are 
developed  for  this  structure. 

5.  Performance  evaluation  of  data  storage  hierarchy 
systems:  Two  GPSS/360  simulation  models  of  the 
INFOPLEX  data  storage  hierarchy  are  developed. 
Simulation  results  reveal  several  unexpected  pro¬ 
perties  of  the  data  storage  hierarchy  design  and 
its  algorithms.  A  well-balanced  system  is  used  to 
compare  the  performance  differential  of  using 
technology  in  1979  versus  projected  technology  in 
1985.  These  simulation  results  indicate  that  the 
current  INFOPLEX  data  storage  hierarchy  design  is 
capable  of  supporting  the  read  and  write  traffic 
generated  by  the  INFOPLEX  functional  hierarchy. 

1.4  STRUCTURE  OF  THESIS 

This  thesis  is  an  important  step  towards  developing  sto¬ 
rage  hierarchy  systems  specifically  for  data  base  computers. 
Existing  models  of  storage  hierarchy  systems  are  extended  to 
obtain  a  formal  model  of  storage  hierarchy  system  which 
incorporates  multiple  page  sizes  and  maintains  multiple  data 
redundancy.  Key  properties  of  such  systems  are  analyzed  in 
detail.  Architectures  of  storage  hierarchy  systems  for 
INFOPLEX  are  developed  and  the  performance  of  these  designs 
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are  evaluated.  Details  of  this  research  are  presented  in 
seven  chapters.  Chapter  one  is  self-explainatory .  The  fol¬ 
lowing  outlines  the  contents  of  the  other  chapters. 


1.4.1  Chapter  2  :  INFOPLEX  Data  Base  Computer 

Architecture 

This  chapter  introduces  the  objectives  and  approaches  of 
the  INFOPLEX  data  base  computer.  Concepts  and  research 
approaches  used  in  the  INFOPLEX  functional  hierarchy  and  the 
INFOPLEX  data  storage  hierarchy  are  described.  This  chapter 
provides  the  background  and  motivation  for  the  research  on 
data  storage  hierarchy  systems. 

1.4.2  Chapter  ^  •  b.  General  Structure  of  the  INFOPLEX 

Data  Storage  Hierarchy 

A  preliminary  design  of  the  INFOPLEX  data  storage  hier¬ 
archy,  DSH-1,  is  proposed.  The  design  objectives  of  DSH-1 
are  discussed.  Then  the  structure  of  DSH-1  is  introduced. 
This  design  can  be  used  to  explore  design  issues  associated 
with  the  INFOPLEX  data  storage  hierarchy.  Key  design  issues 
are  identified. 

1.4.3  Chapter  £  :  Modelling  and  Analysis  of  Data 

Storage  Hierarchy  Systems 

Current  research  efforts  in  storage  hierarchy  systems  are 
briefly  reviewed.  A  formal  model  of  data  storage  hierarchy 
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systems  incorporating  multiple  page  sizes  and  maintain  mul¬ 
tiple  data  redundancy  is  developed.  Extensions  to  the  Least 
Recently  Used  (LRU)  algorithm  are  developed  to  incorporate 
the  read-through  strategy.  Important  performance  and  relia¬ 
bility  properties  of  these  systems  are  formally  proved. 

These  results  provide  valuable  insights  to  designing  data 
storage  hierarchy  systems.  The  formalisms  developed  provide 
a  solid  basis  for  further  theoretic  analysis  of  data  storage 
hierarchy  systems. 

1.4.4  Chapter  5  :  Design  of  the  DSH-11  Data  Storage 

Hierarchy  System 

The  general  structure  of  the  INFOPLEX  data  storage  hier¬ 
archy  is  used  to  derive  a  simpler  structure,  DSH-11.  This 
structure  is  used  as  a  basis  for  developing  protocols  for 
supporting  the  read  and  write  operations.  Specifications 
for  these  protocols  are  presented. 

1.4.5  Chapter  ^  :  Simulation  Studies  of  the  DSH-11  Data 

Storage  Hierarchy  System 

A  simulation  model  of  DSH-11  with  one  processor  and  three 
storage  levels  is  developed.  Results  from  simulation  stu¬ 
dies  using  this  model  provide  valuable  insights  to  the 
DSH-11  design  and  its  algorithms.  This  knowledge  is  incor¬ 
porated  into  another  simulation  model  of  DSH-11  that  con¬ 
sists  of  five  processors  and  four  storage  levels.  Simula- 
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tion  studies  from  this  model  reveal  further  interesting 
properties  of  the  read-through  and  store-behind  algorithms. 
The  simulation  results  also  indicate  that  the  current  design 
is  capable  of  supporting  the  very  high  rate  of  storage 
references  generated  by  the  INFOPLEX  functional  hierarchy. 

1.4.6  Chapter  2  •  Discussions  and  Conclusions 

Chapter  7  summarizes  this  thesis  and  indicates  fruitful 
areas  for  further  research. 
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Chapter  II 

THE  INFOPLEX  DATA  BASE  COMPUTER  ARCHITECTURE 

2.1  INTRODUCTION 

This  chapter  discusses  the  INFOPLEX  data  base  computer 
concepts  and  its  approaches.  Specific  areas  of  contribution 
of  this  thesis  to  the  development  of  the  INFOPLEX  data  base 
computer  are  then  listed. 

The  key  concepts  of  the  INFOPLEX  architecture  are  hier¬ 
archical  decomposition  and  distributed  control.  Techniques 
of  hierarchical  decomposition  are  applied  to  organize  the 
information  management  functions  to  obtain  a  highly  modular 
functional  hierarchy.  Each  level  of  the  functional  hier¬ 
archy  is  implemented  using  multiple  microprocessors.  Tech¬ 
niques  of  hierarchical  decomposition  are  also  applied  to 
organize  the  storage  subsystem  to  obtain  a  modular  storage 
hierarchy  capable  of  supporting  the  storage  requirements  of 
the  functional  hierarchy.  Microprocessors  are  used  at  each 
level  of  the  hierarchy  to  implement  the  storage  management 
algorithms  so  the  hierarchy  appears  as  a  very  large,  highly 
reliable,  high  performance  virtual  storage  space. 
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Due  to  the  high  modularity  of  these  organizations,  both 
the  functional  hierarchy  and  the  storage  hierarchy  can  take 
advantage  of  distributed  control  mechanisms.  Each  level  in 
a  hierarchy  only  communicates  with  its  adjacent  levels  and 
each  module  within  a  level  only  communicates  with  its  adja¬ 
cent  modules.  Thus,  no  central  control  mechanism  is  neces¬ 
sary.  Distributed  control  enhances  reliability  since  there 
is  no  single  component  in  the  system  whose  failure  renders 
the  entire  system  inoperative.  Distributed  control  also 
enhances  performance  since  there  is  no  system  bottleneck  as 
would  exist  in  a  centrally  controlled  system. 

A  functionally  decomposed  hierarchy,  implemented  using 
multiple  microprocessors,  can  support  pipeline  processing 
naturally.  That  is,  multiple  requests  for  information  can 
be  at  various  stages  of  processing  at  different  levels  of 
the  hierarchy  simultaneously.  Such  an  architecture  also 
enhances  reliability  since  errors  can  be  isolated  within  a 
level  in  the  hierarchy  thus  simplifying  error  detection  and 
correction. 

Parallel  processing  is  made  possible  by  the  hierarchical 
decomposition  and  implementation  using  multiple  microproces¬ 
sors.  For  example,  there  may  be  several  identical  modules 
that  implement  the  same  function  within  a  level.  All  these 
modules  can  be  simultaneously  operating  on  different 
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requests,  at  the  same  time,  providing  potential  backup  for 
one  another. 

Thus,  the  distributed  control,  pipeline  and  parallel  pro¬ 
cessing  capabilities  of  INFOPLEX  provide  very  high  reliabil¬ 
ity  and  high  performance. 

In  addition  to  providing  high  performance  and  high  relia¬ 
bility,  a  viable  data  base  computer  must  be  able  to  take 
advantage  of  new  technological  innovations.  It  must  be  able 
to  easily  upgrade  to  incorporate  new  algorithms,  e.g.,  a  new 
security  checking  technique,  or  new  hardware  innovations, 
e.g.,  a  new  storage  device.  Due  to  its  modular  structure, 
the  INFOPLEX  functional  hierarchy  can  take  advantage  of  new 
techniques  and  technologies  as  they  are  developed.  The 
INFOPLEX  storage  hierarchy  is  specifically  designed  to  be 
able  to  handle  any  type  of  storage  devices.  Thus  rather 
than  being  specialized  to  a  particular  data  structure,  or 
type  of  storage  device,  INFOPLEX  is  designed  to  adapt  to  the 
changing  application  needs  as  well  as  to  take  advantage  of 
new  technological  innovations. 

2.2  THE  INFOPLEX  FUNCTIONAL  HIERARCHY 

An  information  management  system  performs  a  spectrum  of 
very  complex  functions  in  response  to  user  requests  for 
information.  These  requests  are  often  expressed  using  very 


31 


high  level  languages  and  often  come  from  many  different 
sources  simultaneously.  There  are  many  ways  that  these  com¬ 
plex  functions  can  be  implemented.  The  technique  of  hier¬ 
archical  functional  decomposition  has  been  found  to  be  very 
effective  for  advanced  information  systems  (Donovan  and 
Jacoby,  1975) .  Similar  techniques  have  been  used  success¬ 
fully  in  operating  systems  (Dijkstra,  1968;  Madnick  and 
Donovan,  1974),  basic  file  systems  (Madnick  and  Alsop,  1969; 
Madnick,  1970),  and  a  wide  range  of  complex  systems  (Pattee, 
1973) . 

This  is  the  approach  used  in  INFOPLEX.  The  information 
management  functions  are  systematically  decomposed  into  a 
functional  hierarchy,  referred  to  as  the  INFOPLEX  functional 
decomposition.  The  functional  modules  in  the  hierarchy  are 
then  implemented  using  multiple  microprocessors. 

2.2.1  Rationale  for  Functional  Decomposition 

The  central  idea  underlying  the  hierarchical  functional 
decomposition  approach  involves  decomposing  the  system  into 
a  hierarchy  consisting  of  a  number  of  levels,  such  that  each 
level  interacts  only  with  the  levels  below  it  in  the  hier¬ 
archy.  Proper  selection  of  the  hierarchy  allows  design  or 
operating  problems  that  previously  impacted  the  entire  sys¬ 
tem,  to  be  isolated  to  one  or  a  few  specific  hierarchical 
levels,  and  thereby  more  easily  handled  (Parnas,  1976). 


32 


Isolating  the  information  management  functions  into  mini¬ 
mally  interrelated  modules  facilitates  the  use  of  multiple 
identical  modules  for  performing  the  same  function,  so  that 
reliability  and  parallelism  are  enhanced.  Furthermore,  this 
approach  provides  great  flexibility  in  the  technologies  used 
for  implementating  each  type  of  functional  module.  For 
example,  a  particular  data  structure  may  be  selected  from  a 
spectrum  of  indexing  techniques  for  a  given  module  without 
affecting  the  design  of  other  types  of  modules. 

2.2.2  Example  of  a  Functional  Decomposition 

To  illustrate  the  hierarchical  functional  decomposition 
concept,  a  specific  example  of  a  functional  decomposition  is 
discussed  in  this  section.  Figure  2.1  illustrates  a  plausi¬ 
ble  hierarchical  functional  decomposition.  Each  level  of 
the  functional  hierarchy  is  described  below. 

2. 2. 2.1  Entities  and  Entity  Sets 

At  the  most  fundamental  level,  a  database  system  stores 
information  about  things,  or  entities.  Also,  it  is  usually 
the  case  that  entities  represented  in  a  database  fall  natur¬ 
ally  into  logical  groups,  or  "entity  sets".  The  way  in 
which  a  database  system  (a)  represents  and  stores  informa¬ 
tion  about  entities  themselves,  and  (b)  represents  informa¬ 
tion  about  the  logical  grouping  of  entities  into  entity 
sets,  forms  the  bedrock  architecture  of  the  system. 
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Figure  2.1  An  Example  Functional  Hierarchy 


There  are  many  schemes  available  for  logically  and  physi¬ 
cally  representing  entities  (i.e.,  coding,  storing,  and 
addressing  entities)  and  various  algorithms  for  structuring 
entity  sets.  The  choice  of  implementation  scheme  at  this 
level  affects  the  performance  of  the  entire  system  but  does 
not  affect  how  the  functions  of  the  other  levels  are  imple¬ 
mented. 

2. 2. 2. 2  Binary  Relations 

All  relationships  among  entities  can  be  expressed  in 
terms  of  binary  relationships  between  pairs  of  entities. 

This  functional  level  makes  use  of  the  entity  level  con¬ 
structs  to  provide  a  collection  of  binary  relations  (rela¬ 
tions  between  pairs  of  entity  sets) .  An  element  of  a  binary 
relation  can  be  viewed  as  a  triad,  consisting  of  a  relation 
identifier  plus  two  entities,  each  from  one  of  the  entity 
sets  participating  in  the  binary  relation.  Thus  a  binary 
relation  can  be  viewed  as  a  collection  of  triads  with  the 
same  relation  identifier. 

Perhaps  the  simplest  possible  implementation  of  a  set  of 
binary  relations  would  be  as  a  sequential  file  of  triads, 
for  example, 

(HAS_SALARY_OF  ,  SMITH  ,  1200) 

(HAS  SALARY  OF  ,  JONES  ,  1500) 
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(WORKS_ 

_IN_ 

_DEPT  . 

SMITH  , 

02) 

(WORKS 

IN 

DEPT  , 

JONES  , 

07) 

The  difficulties  with  this  approach  are  manifest;  there  is 
great  data  redundancy  and  thus  waste  of  storage  (the  rela¬ 
tion  identifiers  are  stored  in  each  triad);  insertion  of 
additional  triads  would  either  have  to  be  done  out  of  order, 
or  else  insertions  and  deletions  would  be  extremely  time- 
consuming. 

Triads  could  also  be  stored  as  linked  lists.  Alterna¬ 
tively  hashing  algorithms  could  be  employed  to  locate  any 
triad,  given  two  of  its  three  components.  The  use  of  linked 
lists  can  improve  access  speed  and  reduce  storage  require¬ 
ments.  On  the  other  hand,  the  use  of  hashing  algorithms 
would  provide  extremely  rapid  access,  but  would  be  poorer  in 
terms  of  storage  space  utilization. 

Since  a  database  may  contain  billions  of  triads,  the  log¬ 
ical  and  physical  structures  of  binary  relations  have  seri¬ 
ous  performance  implications.  Many  implementation  schemes 
for  binary  relations  are  possible.  Although  the  choice  of 
these  implementation  schemes  has  various  cost  and  perfor¬ 
mance  implications  it  does  not  affect  how  the  functions  of 
the  next  level  are  implemented. 


2. 2. 2. 3  N-ary  Relations 

Conceptually,  an  n-ary  relation  may  be  thought  of  as  a 
table  of  data,  with  rows  of  the  table  (usually  called 
tuples)  corresponding  approximately  to  records  in  a  tradi¬ 
tional  data  file,  and  columns  (or  domains)  corresponding  to 
fields.  Furthermore,  n-ary  relations  may  be  constructed  out 
of  sets  of  basic  binary  relations.  For  example,  the  degree 
4  relation  EMPLOYEE_DEPT_SALARY_SEX,  for  which  a  typical 
entry  might  be 

(SMITH,  02,  1200,  male) , 

is  semantically  equivalent  to  (i.e.,  contains  the  same 
information  as)  the  three  binary  relations  WORKS_IN_DEPT, 
HAS_SALARY_OF  and  SEX,  as  illustrated  in  Figure  2.2.  We 
could  build  up  n-ary  relation  tuples  out  of  tuple-ids  of 
binary  relations,  as  illustrated  in  Figure  2.3.  In  this 
approach,  the  original  data  entities  (SMITH,  01,  1200, 
male) ,  would  be  stored  in  permanent  binary  relations,  and 
all  other  relations  would  be  constructed  out  of  binary  tuple 
ids.  Tuple  ids,  being  uniform  binary  numbers,  are  easy  and 
efficient  to  manipulate. 


A  number  of  other  implementations  of  n-ary  relations  is 
also  possible.  The  point  is,  however,  that  once  we  have  an 
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(  BINARY  RELATIONS  ) 

Figure  2.2  An  Example  4-ary  Relation 
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Figure  2.3  An  Example  Implementation  of 
N-ary  Relations 
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REUTION  A 

(name)  (age)  (skill)  id 


Figure  2.4  Links  Among  N-ary  Relations 
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efficient  implementation  of  binary  relations,  general  n-ary 
relations  may  be  constructed  in  a  straightforward  fashion 
out  of  the  binary  relations  without  actually  having  to 
retreat  —  conceptually  or  physically  —  back  to  the  level 
of  basic  entities  or  entity  sets.  In  other  words,  n-ary 
relation  functions  (to  manipulate  n-ary  relations)  can  be 
implemented  by  appropriately  combining  binary  relation  func¬ 
tions. 

2. 2. 2. 4  Links  Among  N-ary  Relations 

The  various  n-ary  relations  in  a  typical  database  would 
generally  possess  a  number  of  logical  interconnections.  For 
example,  one  relation  might  contain  data  on  employees  and 
the  skills  each  employee  possesses,  while  another  might 
involve  data  on  departments  and  the  skills  each  department 
requires  to  function.  The  logical  relationship  between  the 
tuples  in  these  relations  could  be  employed  to  extend  the 
database  structure  further,  by  incorporating  a  set  of 
"meta-relations"  for  storing  information  about  such  links 
between  the  regular  n-ary  relations.  The  role  of  the  meta¬ 
relations  would  be  to  identify  related  tuples,  and  to  pro¬ 
vide  some  semantic  information  regarding  the  nature  of  the 
interrelationships.  In  the  example  cited  above,  it  would 
make  sense  to  establish  a  meta-relation  connecting  the 
appropriate  tuples  in  the  original  two  relations  on  the 
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basis  of  "common  skill",  as  shown  in  Figure  2.4.  Under  the 
implementation  approach  illustrated  in  Figure  2.4,  meta-re- 
lations  would  themselves  be  n-ary  relations.  The  only  dif¬ 
ference  between  them  and  regular  n-ary  relations  lies  in  the 
interpretation  of  their  entries.  Therefore,  all  of  the  pre¬ 
viously  designed  mechanisms  for  building  and  managing  n-ary 
relations  could  also  be  used  with  the  meta-relations.  Only 
the  interpretation  of  the  elements  within  these  relations 
would  be  different. 

By  incorporating  linking  information  among  the  different 
n-ary  relations  in  a  database,  either  permanently  or  tempo¬ 
rarily,  directly  into  the  database  structure  itself,  it 
would  be  possible  to  generate  more  complex  systems  that 
would  be  capable  of  presenting  different  interfaces  to  dif¬ 
ferent  users,  depending  on  the  needs  and  objectives  of  the 
users  themselves. 

2. 2. 2. 5  Virtual  Information 

It  is  not  always  necessary,  or  even  desirable,  that  a 
database  contain  all  the  information  that  users  might  wish 
to  access.  Sometimes  data  interrelationships  are  algor¬ 
ithmic  in  nature,  such  that  certain  values  may  be  unambigu¬ 
ously  derived  from  others  that  are  already  stored  in  the 
database.  This  gives  rise  to  the  concept  of  "virtual" 
information  (Folinus  et  al.,  1974). 
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If  an  employee's  BIRTH_DATE  is  stored  in  a  database,  and 
the  CURRENT_DATE  is  also  available,  then  the  employee's  AGE 
could  be  calculated  by  a  simple  algorithm  and  need  not  also 
be  stored.  If  this  is  in  fact  done,  then  the  employee's  AGE 
would  be  an  example  of  "virtual"  data  —  information  that 
appears  (to  the  database  user)  to  be  stored  there,  but  which 
is  not  actually  present  as  an  entity  in  the  database. 

There  are  a  number  of  advantages  to  "virtualizing"  data 
in  a  database.  These  include; 

1.  Greater  accuracy:  for  example,  an  employee's  AGE 
could  be  calculated  as  accurately  as  necessary  if 
included  as  virtual  data,  whereas  it  would  always 
be  somewhat  "old"  if  it  were  simply  stored  as  a 
database  entity; 

2.  Elimination  of  updating;  virtual  data  items  them¬ 
selves  never  need  updating; 

3.  Reduced  redundancy:  including,  for  example, 
BIRTH_DATE,  CURRENT_DATE ,  and  AGE  as  three  sepa¬ 
rate  items  in  a  database  is  redundant,  and  incon¬ 
sistent  data  relationships  can  easily  result  if 
some  of  the  items  are  updated  independently  of 
others; 
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4.  Savings  in  storage:  in  many  cases,  the  database 
storage  space  required  to  store  items  such  as  AGE 
directly  would  be  much  larger  than  that  required 
to  store  the  coded  algorithm  for  calculating  AGE 
from  other  data. 

One  way  of  implementing  a  virtual  information  capability  is 
to  extend  the  definition  of  n-ary  relations  to  include  tuple 
identifiers  ("ids")  that  would  in  fact  not  refer  to  binary 
relation  tuples,  but  rather  would  point  to  procedures  for 
calculating  the  virtual  data  items.  Consider  a  simple 
employee  relation  of  degree  four,  containing  real  data  items 
NAME,  BIRTH_DATE,  and  SALARY,  plus  a  virtual  data  item  AGE. 
The  organization  of  this  4-tuple  would  then  appear  as  in 
Figure  2.5. 

2. 2. 2. 6  Data  Verification  and  Access  Control 

Data  verification  is  the  process  of  checking  entries  into 
a  database  for  qualities  such  as  reasonableness  (e.g.,  a 
person's  age  should  be  no  greater  than,  say,  125  years),  and 
consistency  (e.g.,  the  sum  of  the  months  worked  in  various 
departments  by  an  employee  should  sum  to  the  number  of 
months  worked  for  the  company) .  Access  control  is  the  pro¬ 
cess  of  controlling  the  database  with  regard  to  data 
retrieval,  update,  deletion,  database  reorganization,  etc. 
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Figure  2.5  Representation  of  Virtual  Information 
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Figure  2.6  An  Example  Data  Verification  Scheme 


For  example,  department  managers  may  be  granted  authoriza¬ 
tion  to  view  the  employee  records  of  only  the  employees 
working  in  their  own  departments;  the  database  administra¬ 
tor,  on  the  other  hand,  may  have  access  to  all  the  records 
in  the  database.  The  database  administrator  may  also  be  the 
only  person  with  authority  to  reorganize  the  entire  data¬ 
base  . 

Access  control  also  involves  considerations  such  as  the 
identification  of  valid  users  through  use  of  passwords  and 
other  such  techniques,  mechanisms  for  allowing  users  to  spe¬ 
cify  the  type  of  access  (read  only,  read/write,  execute 
only,  etc.)  for  files,  and  allowing  users  to  segment  files, 
so  as  to  restrict  parts  of  interconnected  programs  or  data 
files  from  certain  kinds  of  access  by  certain  specified 
users  (an  example  of  a  system  that  has  implemented  this  suc¬ 
cessfully  is  the  MULTICS  system) . 

Both  data  validity  and  access  control  could  be  imple¬ 
mented  in  the  hierarchical  structure  being  discussed  here  in 
a  variety  of  ways.  For  example,  the  basic  n-ary  relations 
could  be  further  extended  to  include  special  control  and 
verification  tuples.  If  data  verification  were  to  be  per¬ 
formed  upon  data  entries  in  a  certain  domain  of  a  relation, 
that  domain  could  be  flagged  in  a  "verification  tuple",  and 
a  data  verification  routine  would  be  called  upon  data  inser- 
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tion  or  update  to  check  the  appropriateness  of  each  entry 
(see  Figure  2.6). 

Similarly,  control  of  access  to  various  domains  or  tuples 
could  be  performed  by  setting  control  bits  in  a  special  con¬ 
trol  tuple  or  domain,  and  including,  for  example,  an  address 
pointer  to  a  list  of  authorized  user  passwords,  against 
which  the  current  user  could  be  checked.  These  control 
tuples  or  flag  bits  would  serve  to  describe  certain  "views", 
or  combinations  of  data  elements,  that  each  user  would  be 
permitted  to  access.  Alternately,  they  could  be  used  to 
describe  elements,  domains,  tuples,  or  entire  relations  that 
a  user  was  not  permitted  to  view. 

Note  that  these  implementations  would  utilize  the  mechan¬ 
isms  employed  to  provide  virtual  information  as  discussed 
above  (i.e.,  certain  ids  are  used  to  point  to  verification 
procedures,  as  they  pointed  to  "virtual  information  computa¬ 
tion  procedures"  in  the  preceding  section).  Thus,  the  veri¬ 
fication  and  access  control  functions  can  be  realized  in 
terms  of  those  responsible  for  virtual  information. 

2.  2.  2. 7  High-level  Language  Interface 

The  user  interface,  through  the  data  manipulation  lan¬ 
guage,  basically  specifies  the  way  in  which  the  database  may 
be  accessed  by  the  users.  In  this  regard,  there  are  three 
main  approaches  to  manipulating  a  database,  corresponding 
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roughly  to  the  three  basic  models  of  database  organization 
(network,  hierarchical,  and  relational.): 

1.  An  application  programmer  may  wish  to  'navigate' 
(Bachman,  1975;  Codasyl,  1971)  a  database  by  using 
the  data  manipulation  language  to  trace  through 
the  data  groupings  (relations)  and  interconnecting 
linkages  (links  between  n-ary  relations) .  This 
approach  to  database  manipulation  is  usually  more 
complex  than  some  others,  and  demands  a  greater 
sophistication  on  the  part  of  the  applications 
programmer.  He  must,  for  example,  be  fully  aware 
of  the  existence  of  all  the  links  connecting  the 
various  data  groupings,  whereas  this  knowledge  is 
not  necessarily  demanded  of  programmers  using 
other  data  manipulation  languages.  In  return  for 
the  greater  complexity,  the  navigational  approach 
usually  offers  greater  accessing  efficiency  and 
better  overall  database  manipulation  performance, 
especially  when  dealing  with  large  and  complex 
databases. 

2.  A  user  may  wish  to  organize  and  manipulate  the 
database  as  a  hierarchical  tree  structure,  wherein 
the  logical  interconnections  between  data  group- 
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ings  are  always  one-to-many  in  nature.  In  a 
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sense,  the  manipulation  of  a  hierarchical  tree 
structure  is  a  special  case  of  the  general  naviga¬ 
tional  approach.  Hierarchical  structures  do,  how¬ 
ever,  allow  a  number  of  simplifications  to  be  made 
in  designing  the  database  management  system,  as 
well  as  in  the  data  manipulation  language.  Furth¬ 
ermore,  a  surprisingly  large  number  of  situations 
in  the  real  world  may  be  effectively  represented 
with  a  hierarchical  tree  data  organization,  so  it 
is  worthwhile  to  treat  hierarchical  structure  as 
an  important  special  case. 

3.  Finally,  in  many  cases  it  is  appropriate  for  the 
applications  programmer  to  access  the  database 
directly  in  terms  of  its  underlying  binary  or 
n-ary  relations  (Codd,  1970;  Codd,  1974).  Such 
"direct"  manipulation  may  be  made  at  a  relatively 
low  level,  in  terms  of  individual  relations  and 
primitive  operations  (using  the  relational  alge¬ 
bra)  upon  them.  Alternately,  a  higher-level 
interface  could  be  used  to  translate  more  general- 
purpose  commands  (using  the  relational  calculus) 
into  lower-level  operations.  Such  low-level 
accessing  methods  generally  provide  greater  effi¬ 
ciency,  at  the  expense  of  greater  programming 
detail . 
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2.2.3  INFOPLEX ' s  Approach  to  Functional  Decomposition 

The  above  discussions  illustrate  one  possible  decomposi¬ 
tion  of  the  information  management  functions  into  hierarchi¬ 
cal  levels.  Other  decompositions  are  possible.  For  exam¬ 
ple,  the  work  of  (Senko,  1976;  Yeh  ^  r  1977;  Toh  e^  al . , 
1977;  ANSI,  1975)  also  decomposes  the  various  information 
management  functions  into  several  levels  (e.g.,  (1)  physical 
data  storage,  (2)  logical  data  encoding,  (3)  access  path, 

(4)  internal  schema,  and  (5)  external  schema).  A  common 
weakness  of  these  functional  decompositions,  including  our 
example  decompositon ,  is  that  although  any  particular  decom¬ 
position  may  make  good  sense  and  impose  a  reasonable  concep¬ 
tual  structure  on  the  information  management  function,  there 
are  no  commonly  accepted  criteria  with  which  to  evaluate  any 
given  decomposition. 

A  common  qualitative  criteria  often  used  to  decompose 
complex  functions  into  sub-modules  is  that  of  modularity.  A 
decomposition  is  considered  to  attain  high  modularity  when 
each  individual  module  is  internally  coherent,  and  all  the 
modules  are  loosely  coupled  with  one  another.  One  of  the 
INFOPLEX  research  focuses  is  to  develop  methodologies  to 
formalize  this  notion  of  modularity  quantitatively,  and  to 
use  it  to  evaluate  a  given  decomposition,  thus  systematic 
techniques  for  obtaining  an  optimal  functional  decomposition 
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of  the  information  management  functions  can  be  developed.  A 
particularly  promising  approach  for  this  purpose  is  the 
Systematic  Design  Methodology  (Huff  and  Madnick,  1978). 

The  following  briefly  describes  this  approach. 

The  Systematic  Design  Methodology  approach  to  system 
design  centers  on  the  problem  of  identifying  a  system's 
modules,  or  "sub-problems",  their  functions,  and  their 
interconnections.  Using  this  approach,  we  begin  with  a  set 
of  functional  requirement  statements  for  the  INFOPLEX  infor¬ 
mation  management  functions.  Each  pair  of  requirements  is 
examined  in  turn,  and  a  decision  as  to  whether  a  significant 
degree  of  interdependence  between  the  two  requirements 
exists  is  made.  Then  the  resulting  information  is  repre¬ 
sented  as  a  non-directed  graph  structure:  nodes  are 
requirement  statements,  links  are  assessed  interdependen¬ 
cies.  The  graph  is  then  partitioned  with  the  objective  of 
locating  a  good  decomposition.  An  index  of  partition  good¬ 
ness  is  employed,  which  incorporates  measures  of  subgraph 
"strength"  and  "coupling".  The  actual  goodness  index  is 
taken  as  the  algebraic  difference  between  the  strengths  of 
all  the  subgraphs,  and  the  inter-subgraph  couplings.  That 
is,  M=S-C,  where  S  is  the  sum  of  the  strength  measures  of 
all  subgraphs,  and  C  is  the  sum  of  all  the  inter-subgraph 
couplings. 
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Once  an  agreeable  partition  is  determined,  the  resulting 
sets  of  requirements  are  interpreted  as  "design  sub-prob¬ 
lems".  From  these  design  sub-problems  a  functional  hier¬ 
archy  of  INFOPLEX  can  then  be  systematically  derived.  This 
procedure  is  illustrated  in  Figure  2.7.  This  approach  is 
currently  being  developed  in  the  INFOPLEX  Project. 

2.3  THE  INFOPLEX  DATA  STORAGE  HIERARCHY 

To  provide  a  high  performance,  highly  reliable,  and  large 
capacity  storage  system,  INFOPLEX  makes  use  of  an  automati¬ 
cally  managed  memory  hierarchy  (referred  to  as  the  INFOPLEX 
physical  decomposition) .  In  this  section,  the  rationale  for 
and  an  example  of  an  automatic  memory  hierarchy  are  dis¬ 
cussed.  Then  the  INFOPLEX  approach  to  realize  such  a  memory 
hierarchy  is  also  discussed. 

2.3.1  Rationale  for  a  Storage  Hierarchy 

The  technologies  that  lend  themselves  to  low  cost-per- 
byte  storage  devices  (and,  thereby,  economical  large  capac¬ 
ity  storage)  result  in  relatively  slow  access  times.  If  it 
was  possible  to  produce  ultra-fast  limitless-capacity  sto¬ 
rage  devices  for  miniscule  cost,  there  would  be  little  need 
for  a  physical  decomposition  of  the  storage.  Lacking  such  a 
wondrous  device,  the  requirements  of  high  performance  at  low 
cost  are  best  satisfied  by  a  mixture  of  technologies  combin- 
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ing  expensive  high-performance  devices  with  inexpensive 
lower-performance  devices. 

There  are  many  ways  that  such  an  ensemble  of  storage  dev¬ 
ices  may  be  structured,  but  the  technique  of  hierarchical 
physical  decomposition  has  been  found  to  be  very  effective 
(Madnick,  1973;  Madnick,  1975a;  Madnick,  1975b).  Using  this 
technique,  thp  ensemble  of  heterogeneous  storage  devices  is 
organized  as  a  hierarchy.  Information  is  moved  between  sto¬ 
rage  levels  automatically  depending  upon  actual  or  antici¬ 
pated  usage  such  that  the  information  most  likely  to  be 
referenced  in  the  future  is  kept  at  the  highest  (most  easily 
accessed)  levels. 

The  effectiveness  of  a  memory  hierarchy  depends  heavily 
on  the  phenomenon  known  as  locality  of  reference  (Denning, 
1970).  A  memory  hierarchy  makes  use  of  this  property  of 
information  reference  pattern  so  that  the  information  that 
is  used  frequently  would  be  accessible  through  the  higher 
levels  of  the  hierarchy,  giving  the  memory  hierarchy  an 
expected  access  time  close  to  that  of  the  access  time  of  the 
faster  memories.  This  approach  has  been  used  in  contempo¬ 
rary  computer  systems  in  cache  memory  systems  (Conti,  1969), 
in  virtual  memory  demand  paging  systems  (Bensoussan  et  al . , 
1969;  Greenberg  and  Webber,  1975;  Hatfield,  1972;  Mattson  ^ 
al. ,  1970;  Meade,  1970),  and  in  mass  storage  systems  (Consi- 
dine  and  Weis,  1969;  Johnson,  1975). 
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Experimentations  with  physical  data  reference  strings  are 
reported  in  (Easton,  1978;  Rodr iguez-Rosell ,  1976;  Smith, 
1978b) .  It  has  been  found  that  there  is  considerable 
sequentiality  of  physical  database  reference  in  these  stu¬ 
dies.  Sequentiality  of  references  is  a  special  form  of  spa¬ 
tial  locality  as  discussed  by  (Madnic^c,  1973).  Several  mea¬ 
sures  of  logical  database  locality  and  experimentations  with 
these  measures  are  reported  in  (McCabe,  1978;  Robidoux, 
1979).  The  observations  from  these  experiments  are  encour¬ 
aging.  In  particular  they  indicate  that  there  is  considera¬ 
ble  locality  of  database  reference. 

2.3.2  Example  of  a  Physical  Decomposition 

We  now  discuss  an  example  of  a  memory  hierarchy,  its  gen¬ 
eral  structure,  types  of  storage  devices  that  it  may  employ, 
and  some  strategies  for  automatic  information  movement  in 
the  hierarchy. 

2. 3. 2.1  General  Structure 

To  the  user  (i.e.  the  lowest  level  of  the  functional 
hierarchy)  of  the  memory  hierarchy,  the  memory  appears  as  a 
very  large  linear  virtual  address  space  with  a  small  access 
time.  The  fact  that  the  memory  is  actually  a  hierarchy  or 
that  a  certain  block  of  information  can  be  obtained  from  a 
certain  level  is  hidden  from  the  memory  user.  Figure  2.8 
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illustrates  the  general  structure  of  a  memory  hierarchy 
consisting  of  six  levels  of  storage  devices.  Some  of  the 
devices  that  can  be  used  in  these  levels  are  discussed  in 
the  next  subsection. 

The  lowest  level  always  contains  all  the  information  of 
the  system.  A  high  level  always  contains  a  subset  of  the 
information  in  the  next  lower  level.  To  satisfy  a  request, 
the  information  in  the  highest  (most  easily  accessed)  level 
is  used. 

Storage  reference  is  accomplished  by  supplying  the  memory 
hierarchy  with  a  virtual  address  (say  a  64-bit  address) ,  the 
memory  hierarchy  will  determine  where  the  addressed  informa¬ 
tion  is  physically  located.  The  addressed  information  will 
be  moved  up  the  memory  hierarchy  if  it  is  found  in  other 
than  the  highest  level  of  the  hierarchy.  This  implies  that 
there  is  a  high  variance  in  the  access  time  of  the  memory 
system.  This  situation  is  alleviated  by  providing  multiple 
ports  to  the  memory  system  so  that  a  pipeline  of  requests 
can  be  processed.  Furthermore,  the  inherent  parallelism 
within  each  memory  level  and  among  different  memory  levels 
provides  high  throughput  for  the  memory  system  as  a  whole. 
Since  the  functional  levels  are  designed  with  high  parallel¬ 
ism  of  operation  as  one  of  its  major  objectives,  the  proces¬ 
sor  making  the  request  can  take  advantage  of  the  high  memory 
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access  time  variance.  Various  schemes  are  used  to  make  the 


automatic  management  of  the  memory  hierarchy  efficient. 

Some  of  these  strategies  are  discussed  in  a  latter  section. 

2. 3. 2. 2  Storage  Devices 

Traditionally,  computer  direct  access  storage  has  been 
dominated  by  two  fairly  distinct  technologies;  (1)  ferrite 
core  and,  later,  metallic  oxide  semiconductor  (MOS)  LSI 
memories  with  microsecond  access  times  and  relatively  high 
costs,  and  (2)  rotating  magnetic  media  (magnetic  drums  and 
disks)  with  access  time  in  the  range  of  10  to  100  millise-  ■ 
conds  and  relatively  low  costs.  This  has  led  to  the  separa¬ 
tion  between  main  storage  and  secondary  storage  devices. 

Recently  several  new  memory  technologies,  most  notably 
magnetic  bubbles,  electron  beam  addressed  memories  (BEAM) , 
and  charge  coupled  devices  (CCD) ,  have  emerged  to  fill  the 
"gap"  between  the  two  traditional  memory  technologies. 

The  evolution  and  increasing  deployment  of  the  above  and 
many  other  memory  technologies  have  produced  a  more  continu¬ 
ous  cost-performance  range  of  storage  devices,  as  depicted 
in  Figure  2.9  (Madnick,  1975a).  Note  that  these  technolo¬ 
gies,  which  are  arbitrarily  grouped  into  six  categories, 
result  in  storage  devices  that  span  more  than  six  orders  of 
magnitude  in  both  random  access  time  (from  less  than  100 
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nanoseconds  to  more  than  1  second)  and  system  price  per  byte 
(from  more  than  50  cents  per  byte  to  less  than  0.0005  cent). 

This  evolution  has  facilitated  the  choice  of  appropriate 
cost-effective  storage  devices  for  the  memory  hierarchy. 

For  example,  for  the  memory  hierarchy  discussed  in  the  pre¬ 
vious  section,  we  might  use  a  device  like  the  IBM  3850  Mass 
Storage  as  the  mass  storage,  traditional  moving  head  disks 
as  secondary  storage,  magnetic  drums  as  backing  store,  CCD 
or  magnetic  bubble  as  block  store,  core  or  semiconductor  RAM 
as  main  storage,  and  high  performance  semiconductor  RAM  as 
cache. 

2. 3. 2. 3  Strategies  for  Information  Movement 

Various  physical  storage  management  and  movement  techni¬ 
ques,  such  as  page  splitting,  read  through,  and  store 
behind,  can  be  distributed  within  the  memory  hierarchy. 

This  facilitates  parallel  and  asynchronous  operation  in  the 
hierarchy.  Furthermore,  these  approaches  can  lead  to 
greatly  increased  reliability  of  operation.  For  example, 
under  the  read  through  strategy  (Figure  2.10) ,  when  data 
currently  stored  at  level  i  (and  all  lower  performance  lev¬ 
els)  is  referenced,  it  is  automatically  and  simultaneously 
copied  and  stored  into  all  higher  performance  levels.  The 
data  itself  is  moved  between  levels  in  standard  transfer 
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units ,  also  called  pages ,  whose  size  N  (i-1,  i)  depends  upon 
the  storage  level  from  which  it  is  being  moved. 

For  example,  suppose  that  the  datum  "a",  at  level  3,  is 
referenced  (see  Figure  2.10).  The  block  of  size  N{2,3)  con¬ 
taining  "a"  is  extracted  and  moved  up  the  data  bus.  Level  2 
extracts  this  block  of  data  and  stores  it  in  its  memory 
modules.  At  the  same  time,  level  1  extracts  a  sub-block  of 
size  N{1,2)  containing  "a"  and  level  0  extracts  the  sub¬ 
block  of  size  N(0,1)  containing  "a"  from  the  data  bus. 

Hence,  under  the  read  through  strategy,  all  upper  storage 
levels  receive  this  information  simultaneously.  If  a  sto¬ 
rage  level  must  be  removed  from  the  system,  there  are  no 
changes  needed.  In  this  case,  the  information  is  "read 
through"  the  level  as  if  it  didn't  exist.  Since  all  data 
available  at  level  i  is  also  available  at  level  i  +  1  (and 
all  other  lower  performance  levels) ,  there  is  no  information 
lost.  Thus,  no  changes  are  needed  to  any  of  the  other  sto¬ 
rage  levels  or  the  storage  management  algorithms  although  we 
would  expect  the  performance  to  decrease  as  a  result  of  the 
missing  storage  level.  A  limited  form  of  this  reliability 
strategy  is  employed  in  most  current-day  cache  memory  sys¬ 
tems  (Conti ,  1969) . 
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In  a  store  behind  strategy  all  information  to  be  changed 
is  first  stored  in  L(l),  the  highest  performance  storage 
level.  This  information  is  marked  "changed"  and  is  copied 
into  L(2)  as  soon  as  possible,  usually  during  a  time  when 
there  is  little  or  no  activity  between  L(l)  and  L{2).  At  a 
later  time,  the  information  is  copied  from  L(2)  to  L(3), 
etc.  A  variation  on  this  strategy  is  used  in  the  MULTICS 
Multilevel  Paging  Hierarchy  (Greenberg  and  Webber,  1975). 
This  strategy  facilitates  more  even  usage  of  the  bus  between 
levels  by  only  scheduling  data  transfers  between  levels  dur¬ 
ing  idle  bus  cycles.  Furthermore,  the  time  required  for  a 
write  is  only  limited  by  the  speed  of  the  highest  level 
memory. 

The  store  behind  strategy  can  be  used  to  provide  high 
reliability  in  the  storage  system.  Ordinarily,  a  changed 
page  is  not  allowed  to  be  purged  from  a  storage  level  until 
the  next  lower  level  has  been  updated.  This  can  be  extended 
to  require  two  levels  of  acknowledgment.  Under  such  a  stra¬ 
tegy,  a  changed  page  cannot  be  removed  from  L(l)  until  the 
corresponding  pages  in  both  L(2)  and  L(3)  have  been  updated. 
In  this  way,  there  will  be  at  least  two  copies  of  each 
changed  piece  of  information  at  levels  L{i)  and  L(i+1)  in 
the  hierarchy.  Other  than  slightly  delaying  the  time  at 
which  a  page  may  be  purged  from  a  level,  this  approach  does 
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not  significantly  affect  system  performance.  As  a  result  of 
this  technique,  if  any  level  malfunctions,  it  can  be  removed 
from  the  hierarchy  without  causing  any  information  to  be 
lost.  There  are  two  exceptions  to  this  process,  L(l)  and 
L(n).  To  completely  safeguard  the  reliability  of  the  sys¬ 
tem,  it  may  be  necessary  to  store  duplicate  copies  of  infor¬ 
mation  at  these  levels  only. 

Figure  2.11  illustrates  this  process.  In  Figure  2.11(a), 
a  processor  stores  into  L(l),  the  corresponding  page  is 
marked  "changed"  and  "no  lower  level  copy  exists".  Figure 
2.11(b)  shows  in  a  latter  time,  the  corresponding  page  in 
L(2)  is  updated  and  marked  "changed"  and  "no  lower  level 
copy  exists".  An  acknowledgment  is  sent  to  L(l)  so  that  the 
corresponding  page  is  marked  "one  lower  level  copy  exists". 
At  a  later  time  (Figure  2.11(c)),  the  corresponding  page  in 
L(3)  is  updated  and  marked  "changed"  and  "no  lower  level 
copy  exists".  An  acknowledgment  is  sent  to  L(2)  so  that  the 
corresponding  page  is  marked  "one  lower  level  copy  exists". 
An  acknowledgment  is  sent  to  L(l)  so  that  the  corresponding 
page  is  marked  "two  lower  level  copy  exists".  At  this  time, 
the  page  in  L(l)  may  be  replaced  if  necessary,  since  then 
there  will  be  at  least  two  copies  of  the  updated  information 
in  the  lower  memory  levels. 
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Figure  2.11(a)  Store  Behind  (a) 
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Figure  2.11(b)  Store  Behind  (b) 
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Figure  2.11(c)  Store  Behind  (c) 
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2,3.3  INFOPLEX ' s  Approach  to  Physical  Decomposition 

In  the  previous  section,  we  have  illustrated  an  example 
of  a  memory  hierarchy  that  makes  use  of  an  ensemble  of  het¬ 
erogeneous  storage  devices.  Although  memory  hierarchies 
using  two  or  three  levels  of  storage  devices  have  been 
implemented,  no  known  generalized  automatic  memory  hierarchy 
has  been  developed. 

The  optimality  of  a  memory  hierarchy  depends  on  the  com¬ 
plex  interactions  among  the  memory  reference  pattern,  the 
device  characteristics,  and  the  information  movement  strate¬ 
gies.  The  INFOPLEX  approach  to  this  complex  problem  is  to 
empirically  study  and  characterize  data  reference  patterns 
at  several  levels  (e.g.  transaction  level,  logical  data 
level,  and  physical  data  level) ,  to  develop  various  informa¬ 
tion  movement  strategies,  and  to  design  a  prototype  memory 
hierarchy.  The  interactions  among  these  components  can  then 
be  systematically  investigated  by  means  of  analytic  models 
and  simulation  models. 

2.4  RESEARCH  ISSUES  ADDRESSED  ^  THIS  THESIS 

This  chapter  has  provided  the  background  for  this  thesis. 
As  is  evident  from  the  above  discussions,  there  are  a  large 
number  of  interesting  but  unresolved  research  problems  asso¬ 
ciated  with  INFOPLEX.  This  thesis  is  a  key  step  towards 
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understanding  the  INFOPLEX  data  storage  hierarchy.  In  par¬ 
ticular,  this  thesis  has  made  contributions  in  the  following 
areas : 

1.  Developed  and  extended  concepts  for  the  INFOPLEX 
data  base  computer  and  data  storage  hierarchy  sys¬ 
tems  . 

2.  Provided  a  theoretic  foundation  for  analysis  of 
data  storage  hierarchy  systems. 

3.  Formalized  storage  management  algorithms  to  incor¬ 
porate  the  read-through  strategy. 

4.  Provided  detail  analysis  of  the  performance  and 
reliability  properties  of  data  storage  hierarchies 
and  their  algorithms. 

5.  Designed  prototype  data  storage  hierarchy  systems 
for  INFOPLEX. 

6.  Developed  simulation  models  to  obtain  insights  to 
the  data  storage  hierarchy  designs  and  their 
algor ithras . 

These  are  elaborated  in  the  following  chapters. 
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Chapter  III 

A  GENERAL  STRUCTURE  OF  THE  INFOPLEX  DATA  STORAGE  HIERARCHY 

3.1  INTRODUCTION 

This  chapter  proposes  the  general  structure  of  a  data 
storage  hierarchy  system  for  the  INFOPLEX  data  base  compu¬ 
ter.  The  design  of  this  system  is  based  on  Madnick's  pro¬ 
posed  system  (Madnick,  1973).  This  work  brings  Madnick's 
system  one  step  closer  to  realization.  In  the  following, 
the  underlying  design  goals  of  this  data  storage  hierarchy 
system  will  be  discussed.  Then  the  design,  called  DSH-1,  is 
introduced  followed  by  a  discussion  of  further  design  issues 
that  need  to  be  addressed. 

3.2  DESIGN  OBJECTIVES 

There  are  a  large  number  of  practical  storage  hierarchy 
systems  today.  However,  the  functionality  provided  by  each 
is  quite  different  and  often  falls  short  of  our  expectations 
(for  use  as  the  storage  subsystem  of  the  INFOPLEX  data  base 
computer).  In  the  following,  we  discuss  the  underlying 
design  goals  of  DSH-1. 
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3.2.1  Virtual  Address  Space 

DSH-1  provides  a  virtual  address  space  for  data  storage. 
Every  data  item  in  DSH-1  is  byte  addressable  using  a  gener¬ 
alized  virtual  address.  A  key  advantage  of  a  virtual 
address  space  is  that  a  user  (a  processor)  of  DSH-1  is 
relieved  of  all  physical  device  concerns.  In  fact,  the  pro¬ 
cessor  accessing  DSH-1  is  not  aware  of  how  the  virtual 
address  space  is  implemented.  This  latter  characteristic  is 
quite  unique  since  most  current  virtual  memory  systems  are 
simulated,  at  least  partially,  by  software  executed  by  the 
processor,  e.g.,  the  IBM  OS/VS  system  (Scherr ,  1973) . 

3.2.2  Very  Large  Address  Space 

Early  virtual  memory  systems  were  developed  primarily  for 
program  storage,  hence  their  address  spaces  were  quite  lim¬ 
ited,  e.g.,  in  the  order  of  one  million  bytes.  The  MULTICS 
(Greenberg  and  Webber,  1975)  virtual  memory  and  the  IBM  Sys¬ 
tem/38  (Datamation,  1978;  Soltis  and  Hoffman,  1979)  logical 
storage  were  developed  for  program  as  well  as  data  file  sto¬ 
rage.  These  systems  support  a  large  virtual  address  space. 
However,  the  size  of  an  individual  data  file  in  MULTICS  is 
limited  to  2**18  bytes  and  that  in  System/38  is  limited  to 
2**24  bytes.  Though  these  are  very  large  address  spaces,  it 
is  expected  that  future  systems  will  require  online  storage 
capacities  that  are  much  larger.  DSH-1  uses  a  64-bit  vir- 
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tual  address.  Each  byte  is  directly  addressable,  hence 
there  is  virtually  no  limit  on  the  size  of  a  logical  entity 
such  as  a  data  file. 

3.2.3  Permanent  Data  Storage 

Accesses  to  permanent  data  is  performed  by  special  soft¬ 
ware  routines  and  a  special  I/O  processor  in  most  virtual 
memory  systems.  The  I/O  processor  brings  the  data  into  the 
virtual  memory  and  writes  the  data  back  to  permanent  storage 
when  the  data  is  updated.  Systems  like  MULTICS  and  Sys¬ 
tem/38  provide  a  permanent  virtual  data  storage.  Any  data 
in  virtual  memory  is  also  in  permanent  storage.  DSH-1  also 
provides  a  permanent  virtual  data  storage.  Special  data 
integrity  schemes  are  used  to  ensure  that  as  soon  as  a  pro¬ 
cessor  completes  a  write  operation  to  a  virtual  location, 
the  effect  of  the  write  becomes  permanent  even  in  the  event 
of  a  power  failure. 

3.2.4  Support  Multiple  Processors 

Most  current  virtual  memory  systems  have  been  limited  to 
supporting  2  to  3  processors.  It  is  necessary  that  DSH-1 
support  a  large  number  of  processors  due  to  the  requirements 
for  high  performance  and  high  availability  to  be  discussed 
below.  All  these  processors  share  the  same  virtual  data 
address  space.  Appropriate  synchronization  and  protection 
schemes  are  used  to  ensure  data  integrity  and  security. 
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3.2.5  Generalized  Multi-level  Storage  System 

To  provide  a  large  capacity  storage  subsystem  with  low 
cost  and  high  performance,  a  spectrum  of  storage  devices 
arranged  in  a  hierarchy  is  used.  Previous  storage  hierarchy 
systems  have  been  specially  designed  for  a  specific  2  or  3 
levels  hierarchy  (e.g.,  cache  and  main  memory,  or  main 
memory  and  secondary  storage  device).  Thus,  it  is  extremely 
difficult  to  add  or  remove  a  storage  level  in  these  systems. 
DSH-1  is  designed  to  incorporate  any  type  of  storage  device 
and  support  reconfiguration  of  storage  levels.  This  charac¬ 
teristic  is  particularly  important  in  responding  to  new  dev¬ 
ice  technologies. 

3.2.6  Direct  Inter-level  Data  Transfer 

In  most  current  storage  hierarchy  systems,  data  movement 
among  storage  levels  is  performed  indirectly.  For  example, 
to  move  data  from  drum  to  disk  in  the  MULTICS  system,  data 
is  read  from  drum  into  main  memory  by  the  processor  which 
then  writes  the  data  to  disk.  Recent  developments  in  sto¬ 
rage  systems  make  it  possible  to  decentralize  the  control  of 
data  movement  between  storage  devices  to  intelligent  con¬ 
trollers  at  the  storage  devices.  For  example,  the  IBM  3850 
Mass  Storage  (Johnson,  1975)  uses  an  intelligent  controller 
to  handle  data  transfer  between  mass  storage  and  disks,  mak¬ 
ing  the  3850  appear  as  a  very  large  number  of  virtual  disks. 
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DSH-1  incorporates  intelligent  controllers  at  each  storage 
level  to  implement  the  algorithms  for  data  movement  among 
the  storage  levels.  Special  algorithms  are  developed  to 
facilitate  efficient  broadcasting  of  data  from  a  storage 
level  to  all  other  storage  levels  as  well  as  movement  of 
data  between  adjacent  storage  levels. 

3.2.7  High  performance 

To  support  the  data  requirements  of  the  functional  pro¬ 
cessors  in  INFOPLEX,  DSH-1  is  designed  to  handle  a  large 
number  of  requests  simultaneously.  The  operation  of  DSH-1 
is  highly  parallel  and  asychronous.  Thus,  many  requests  may 
be  in  different  stages  of  completion  at  various  storage  lev¬ 
els  of  DSH-1.  Each  processor  accesses  DSH-1  through  a  data 
cache  where  the  most  frequently  used  data  items  are  stored. 

3.2.8  Availability 

High  availability  of  DSH-1  is  a  result  of  a  combination 
of  the  design  strategy  used,  hardware  commonality,  and  spe¬ 
cial  algorithms.  Key  design  strategies  in  DSH-1  include  the 
use  of  distributed  controls  and  simple  bus  structures,  both 
of  which  contribute  to  the  high  availability  of  DSH-1.  Mul¬ 
tiple  identical  hardware  components  are  used  in  parallel  to 
provide  high  performance  and  to  ensure  that  no  single  compo¬ 
nent  is  critical  to  system  operation.  Integrated  into  the 
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design  are  certain  algorithms  that  exploit  the  structure  of 
DSH-1  to  allow  data  redundancy  and  perform  automatic  data 
repair  in  the  event  of  component  failure,  thus  diminishing 
the  dangers  of  multiple  failures. 

3.2.9  Modularity 

DSH-1  is  modular  at  several  levels.  This  provides  much 
flexibility  in  system  structuring.  The  number  of  processors 
to  be  supported  by  DSH-1  can  be  varied.  The  number  of  sto¬ 
rage  levels  and  the  type  of  storage  devices  can  be  chosen  to 
meet  the  particular  capacity  and  performance  requirements. 
All  the  storage  levels  have  very  similar  structures  and  the 
same  algorithm  is  used  by  the  intelligent  controllers  at 
each  storage  level. 

Flexibility  in  system  structuring  is  extended  in  DSH-1  to 
allow  for  dynamic  system  reconfiguration.  For  example,  a 
defective  storage  device  or  storage  level  can  be  amputated 
without  loss  of  system  availability. 

An  example  of  a  system  that  also  incorporates  modularity 
as  a  key  design  goal  is  the  PLURIBUS  (Katsuki  ^. ,  1978) 

system.  In  PLURIBUS,  the  basic  building  block  is  a  bus 
module.  The  number  of  components  on  a  bus  module  as  well  as 
the  number  of  bus  modules  can  be  easily  varied  to  meet  dif¬ 
ferent  system  requirements. 
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3.2.10  Low  Cost 


A  storage  hierarchy  is  the  lowest  cost  configuration  to 
meet  the  requirement  of  providing  a  large  storage  capacity 
with  high  performance.  DSH-1  also  make  use  of  common  hard¬ 
ware  modules  as  the  intelligent  controllers  at  each  storage 
level,  thus  reducing  hardware  development  cost.  The  modu¬ 
larity  features  of  DSH-1  discussed  above  also  facilitate 
system  upgrading  with  minimum  cost. 

Commonality  of  hardware  modules  and  flexibility  of  system 
upgrade  have  been  employed  in  many  computer  systems  as  an 
effective  approach  to  reduce  cost.  However,  these  techni¬ 
ques  are  rarely  applied  to  storage  hierarchy  systems.  DSH-1 
is  a  step  in  this  direction. 

Advances  in  storage  device  and  processor  technologies 
provide  great  potentials  for  development  of  very  effective 
data  storage  hierarchies  that  incorporate  the  above  charac¬ 
teristics.  In  the  next  section,  we  describe  a  general 
structure  of  such  a  system. 

3.3  GENERAL  STRUCTURE  OF  DSH-1 

The  structure  of  DSH-1  is  illustrated  in  Figure  3.1.  A 
key  design  decision  in  DSH-1  is  to  make  use  of  an  asynchro¬ 
nous  time-shared  bus  for  interconnecting  multiple  components 
(processors  and  memory  modules)  within  a  storage  level  and 
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to  make  use  of  an  asynchronous  time-shared  bus  for 
interconnecting  all  the  storage  levels.  A  key  advantage  of 
the  time-shared  bus  is  its  simplicity,  flexibility,  and 
throughput.  Two  alternative  approaches  can  be  used  in  DSH-1 
to  increase  the  effective  bandwidth  of  the  time-shared  bus. 
First,  a  new  pended-bus  protocol  can  be  used  (Haagens, 

1978) .  This  asynchronous  bus  protocol  is  more  efficient 
than  the  usual  time-shared  bus  protocols  with  the  result 
that  a  much  larger  number  of  components  can  share  a  single 
bus.  Second,  multiple  logical  buses  can  be  used  to  parti¬ 
tion  the  load  on  the  time-shared  bus. 

In  the  following  subsections,  we  shall  describe  the 
interface  to  DSH-1  as  seen  by  a  functional  hierarchy  proces¬ 
sor.  Then  the  structure  of  DSH-1  is  described  by  examining 
its  highest  performance  storage  level  and  then  a  typical 
storage  level. 

3.3.1  The  DSH-1  Interface 

To  the  functional  hierarchy  processors  connected  to 
DSH-1,  DSH-1  appears  as  a  large  multi-port  main  memory. 

There  are  K  memory  ports,  hence  K  processors  can  simultane¬ 
ously  access  DSH-1. 

The  functional  processors  use  a  2**V  (V=64)  byte  virtual 
address  space.  The  instructions  for  each  functional  hier- 
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archy  processor  are  stored  in  a  separate  2**1  byte  program 
memory.  The  program  memories  are  not  part  of  DSH-1.  Thus, 
2**1  bytes  of  the  processor's  address  space  is  mapped  by  the 
program  memories,  leaving  2**V-2**I  bytes  of  data  memory  to 
be  managed  by  DSH-1.  This  is  depicted  in  Figures  3.2(a)  and 
3.  2  (b)  . 

Each  processor  has  multiple  register  sets  to  support 
efficient  multiprogramming.  Some  of  the  more  important 
registers  for  interfacing  with  DSH-1  are  :  (1)  a  V-bit 

Memory  Address  Register  (MAR)  for  holding  the  virtual 
address,  (2)  a  Memory  Buffer  Register  (MBR)  for  storing  the 
data  read  from  DSH-1  and  to  be  written  into  DSH-1,  (3)  a 
Memory  Operation  Register  (MOR)  indicates  the  particular 
operation  to  be  performed  by  DSH-1,  (4)  an  Operation  Status 
Register  (OSR)  which  indicates  the  result  of  a  operation 
performed  by  DSH-1,  and  (5)  a  Process  Identifier  Register 
(PIR)  which  contains  the  Process  Identifier  (PID)  of  the 
process  that  is  currently  using  the  processor. 

t 

A  number  of  memory  operations  are  possible.  The  key  ones 
are  the  read  and  write  operations  and  the  primitives  for 
locking  a  data  item  (such  as  those  supporting  the  Test-and- 
Set  type  of  operations) . 


80 


Figure  3.2(a)  The  DSH-1  Interface 
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Figure  3.2(b)  The  DSH-1  Address  Space 
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All  read  and  write  operations  to  DSH-1  are  performed  in 
the  highest  performance  storage  level,  L{1).  If  a  refer¬ 
enced  data  item  is  not  in  L(l),  it  is  brought  up  to  L{1) 
from  a  lower  storage  level  via  a  r ead-through  operation. 

The  effect  of  an  update  to  a  data  item  in  L(l)  is  propagated 
down  to  the  lower  storage  levels  via  a  number  of  s tor e-be- 
hind  operations. 

In  a  read  operation,  two  results  can  occur  depending  on 
the  state  of  DSH-1.  First,  if  the  requested  data  is  already 
in  L{1),  the  MBR  is  filled  with  the  data  bytes  starting  at 
location  (MAR)  and  the  processor  continues  with  the  next 
operation.  Alternatively,  the  addressed  data  may  not  be 
available  in  L(l).  In  this  case,  the  processor  is  inter¬ 
rupted,  the  OSR  is  set  to  indicate  that  it  may  take  a  while 
for  the  read  operation  to  complete,  and  the  processor  is 
switched  to  another  process.  In  the  meantime,  the  addressed 
data  is  copied  into  L(l)  from  a  lower  storage  level.  When 
this  is  completed,  the  processor  is  notified  of  the  comple¬ 
tion  of  the  original  read  operation. 

Similarly,  a  write  operation  may  result  in  two  possible 
responses  from  DSH-1.  First,  if  the  data  to  be  updated  is 
already  in  L(l),  the  bytes  in  MBR  are  written  to  the  virtual 
address  locations  starting  at  (MAR) ,  and  the  processor  con¬ 
tinues  with  the  next  operation.  Second,  a  delay  similar  to 
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the  read  operation  may  occur  (when  the  data  to  be  updated  is 
not  in  L{1)),  while  DSH-1  retrieves  the  data  from  a  lower 
storage  level. 

This  concludes  a  brief  description  of  the  asynchronous 
DSH-1  interface,  as  seen  by  a  functional  hierarchy  proces¬ 
sor.  Next,  we  examine  the  structure  of  DSH-1. 

3.3.2  The  Highest  Performance  Storage  Level  -  L(l^) 

There  are  h  storage  levels  in  DSH-1,  labelled  L{1),  L(2), 
L(3),  ...,  L(h).  L(l)  is  the  highest  performance  storage 
level.  L(i)  denotes  a  typical  storage  level.  The  structure 
of  L(l)  is  unique.  The  structures  of  all  other  storage  lev¬ 
els  are  similar. 

A  distinction  must  be  made  between  the  concept  of  a  phy¬ 
sical  bus  and  a  logical  bus.  The  former  refers  to  the 
actual  hardware  that  implements  communications  among  levels 
and  within  a  level.  A  logical  bus  may  be  implemented  using 
one  or  more  physical  buses.  Logical  buses  represent  a  par¬ 
titioning,  based  upon  the  virtual  address  referenced,  of  the 
physical  buses. 

Referring  to  Figure  3.1,  L(l)  consists  of  K  memory  ports 
and  S(l)  storage  level  controllers  (SLC's)  on  each  of  B(l) 
logical  local  buses  (i.e.,  S(1)*B(1)  SLC's  in  total  for  this 
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level) .  Each  memory  port  consists  of  a  data  cache  control¬ 
ler  (DCC)  and  a  data  cache  duplex  (DCD) .  A  DCC  interfaces 
with  the  functional  hierarchy  processor  that  is  connected  to 
the  memory  port.  A  DCC  also  performs  mapping  of  a  virtual 
address  generated  by  the  processor  to  a  physical  address  in 
the  DCD.  Another  function  of  DCC  is  to  interface  with  other 
Dec's  (e.g.,  to  maintain  data  cache  consistency),  and  with 
SLC's  on  the  logical  bus  (for  communications  with  other  sto¬ 
rage  levels) . 

At  L(l),  a  SLC  accepts  requests  to  lower  storage  levels 
from  the  DCC's  and  forwards  them  to  a  SLC  at  the  next  lower 
storage  level.  When  the  responses  to  these  requests  are 
ready,  the  SLC  accepts  them  and  sends  them  back  to  the 
appropriate  DCC's.  The  SLC's  also  couple  the  local  buses  to 
the  global  buses.  In  essence,  the  SLC  serves  as  a  gateway 
between  levels  and  they  contend  among  themselves  for  use  of 
the  communication  media,  the  logical  buses. 

At  L(l),  there  are  B(l)  logical  local  buses.  Each  logi¬ 
cal  local  bus  consists  of  b(l)  physical  buses.  Each  logical 
bus  handles  a  partition  of  the  addresses.  For  example,  if 
two  logical  buses  were  used,  one  might  handle  all  odd  num¬ 
bered  data  blocks  and  the  other  would  handle  all  the  even 
numbered  data  blocks. 
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DSH-1  has  B(0)  logical  global  buses.  Each  logical  global 
bus  consists  of  b(0)  global  physical  buses.  The  use  of 
address  partitioning  increases  the  effective  bus  bandwidth. 
The  use  of  multiple  physical  buses  for  each  logical  bus 
enhances  reliability  and  performance. 

3.3.3  A  Typical  Storage  Level  -  L{_i) 

A  typical  storage  level,  L(i) ,  is  divided  into  B(i) 
address  partitions.  Each  address  partition  consists  of  S(i) 
SLC's,  P(i)  memory  request  processors  (MRP's),  and  D(i)  sto¬ 
rage  device  modules  (SDM's),  all  sharing  a  logical  bus.  A 
logical  bus  consists  of  b{i)  physical  buses. 

An  SLC  is  the  communication  gateway  between  the 
MRP's/SDM's  of  its  level  and  the  other  storage  levels. 

An  MRP  performs  the  address  mapping  function.  It  con¬ 
tains  a  directory  of  all  the  data  maintained  in  the  address 
partition.  Using  this  directory,  an  MRP  can  quickly  deter¬ 
mine  if  a  virtual  address  corresponds  to  any  data  in  the 
address  partition,  and  if  so,  what  the  real  address  is  for 
the  data.  This  real  address  can  be  used  by  the  correspond¬ 
ing  SDM  to  retrieve  the  data.  Since  each  MRP  contains  a 
copy  of  this  directory,  updates  to  the  directory  have  to  be 
handled  with  care,  so  that  all  the  MRP's  see  a  consistent 
copy  of  the  directory. 
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An  SDM  performs  the  actual  reading  and  writing  of  data. 

It  also  communicates  with  the  MRP's  and  the  SLC's. 

The  SLC's,  MRP's,  and  SDM's  cooperate  to  handle  a  memory 
request.  An  SLC  communicates  with  other  storage  levels  and 
passes  requests  to  an  MRP  to  perform  the  address  transla¬ 
tion.  The  appropriate  SDM  is  then  initiated  to  read  or 
write  the  data.  The  response  is  then  sent  to  another  SLC  at 
another  storage  level. 

3.4  FURTHER  DESIGN  ISSUES 

The  previous  section  describes  the  general  structure  of 
DSH-1.  From  this  general  structure,  a  number  of  interesting 
alternative  configurations  can  be  obtained.  For  example,  if 
all  the  data  caches  are  taken  away,  L(l)  becomes  a  level 
with  only  the  SLC's  for  communicating  the  requests  from  the 
processors  to  the  lower  storage  levels  and  for  obtaining 
responses  from  these  lower  storage  levels.  This  configura¬ 
tion  eliminates  the  data  consistency  problems  associated 
with  multiple  data  caches. 

If  we  let  the  number  of  logical  buses  be  equal  to  one,  we 
obtain  the  configuration  without  address  partitioning. 

Another  intersting  configuration  is  when  there  is  only 
one  MRP  and  one  SDM  on  a  given  logical  bus.  This  configura- 
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tion  eliminates  the  need  for  multiple  identical  directory 
updates . 

Thus,  by  varying  the  design  parameters  of  DSH-1,  a  large 
number  of  alternative  configurations  with  quite  different 
characteristics  can  be  obtained.  The  general  structure  is  a 
valuable  vehicle  for  investigating  various  design  issues. 
Some  of  the  key  issues  are  briefly  introduced  in  the  follow¬ 
ing  sections. 

3.4.1  Support  of  Read  and  Write  Operations 

Key  problems  in  supporting  the  read  and  write  operations 
in  DSH-1  include  :  (1)  data  consistency  in  multiple  data 

caches,  (2)  protocols  for  communicating  over  the  shared 
bus,  (3)  algorithms  for  updating  the  redundant  directories, 
(4)  algorithms  for  arbitrating  among  usage  of  identical 
resources,  such  as  buses,  SLC's  and  MRP's,  and  (5)  specify¬ 
ing  the  various  steps  (transactions)  that  have  to  be  accom¬ 
plished  to  handle  the  read  and  write  operations. 

3. 4. 1.1  Multiple  Cache  Consistency 

As  illustrated  in  Figure  3.1,  each  DSH-1  memory  port  is  a 
data  cache  directly  addressable  by  the  processor  at  the 
port.  It  is  possible  then,  that  a  data  item  may  be  in  sev¬ 
eral  different  data  caches  at  the  same  time.  When  the  data 
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item  gets  updated  by  a  processor,  other  processors  may 
reference  an  inconsistent  copy  of  the  data  item.  The  multi¬ 
ple  cache  consistency  problem  and  its  solutions  are  dis¬ 
cussed  in  (Tang,  1976;  Censier  and  Feautrier,  1978). 

Three  basic  approaches  can  be  used  to  resolve  this  prob¬ 
lem  in  DSH-1.  The  first  approach  is  to  send  a  purge  request 
to  all  other  data  caches  whenever  a  processor  updates  data 
in  its  cache.  The  second  approach  is  to  maintain  status 
information  about  the  data  cache  contents.  Whenever  there 
is  an  update  to  a  data  item,  this  status  information  is  con¬ 
sulted  and  purge  requests  are  sent  only  to  those  caches  that 
contain  the  data  item  being  changed.  The  third  approach  is 
to  make  use  of  knowledge  of  how  the  data  in  DSH-1  is  to  be 
used  so  that  the  inconsistency  problem  can  be  avoided.  For 
example,  knowledge  about  the  interlocking  scheme  used  to 
ensure  safe  data  sharing  may  be  used  to  avoid  uncessary 
purge  requests  to  other  caches. 

3. 4. 1,2  Bus  Communication  Protocols 

In  DSH-1,  the  buses  may  be  used  for  point-to-point  commu¬ 
nication  as  well  as  for  broadcast  type  of  communications. 

It  is  necessary  to  ensure  that  messages  are  sent  and 
received  correctly.  For  example,  L(i)  broadcast  data  to  the 
upper  levels  and  one  or  more  of  these  levels  may  not  be  able 
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to  accomodate  the  data  to  be  received,  possibly  due  to  the 
lack  of  buffer  space.  Communications  protocols  to  handle 
these  situations  are  important. 

3. 4. 1.3  Multiple  Directory  Update 

Each  MRP  contains  a  directory  of  all  the  data  in  the 
SDM's  on  the  same  bus.  Multiple  requests  may  be  handled  by 
the  MRP's.  When  a  MRP  updates  its  directory,  other  MRP's 
may  still  reference  the  old  copy  of  the  directory.  This  is 
similar  but  not  identical  to  the  multiple  cache  consistency 
problem  discussed  above.  It  is  necessary  to  maintain  con¬ 
sistency  of  the  MRP  directory  states. 

3. 4. 1.4  Multiple  Resource  Arbitration 

Multiple  identical  resources  (e.g.,  buses,  MRP's,  and 
SLC's)  are  used  in  DSH-1  to  provide  parallel  processing 
while  at  the  same  time  providing  redundancy  against  failure. 
A  request  for  a  resource  can  be  satisfied  by  any  one  of  the 
resources.  An  arbitration  scheme  is  required  to  control  the 
assignment  of  resource. 

3. 4. 1.5  Transaction  Handling 

A  read  or  a  write  request  may  go  through  a  number  of 
asynchronous  steps  through  a  number  of  storage  levels  to 
completion.  A  complication  to  these  transactions  is  that 
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for  high  throughput,  a  request  (or  response)  may  be  divided 
into  a  number  of  messages  when  the  request  (or  response)  is 
being  transported  within  the  hierarchy.  Thus,  a  request  (or 
response)  may  have  to  be  assembled,  which  may  take  an  amount 
of  time  dependent  on  the  traffic  within  DSH-1.  Partial 
requests  (responses)  at  a  storage  level  require  special  han¬ 
dling. 

3.4.2  Multiple  Data  Redundancy  Properties 

As  a  result  of  the  read-through  operation,  several  copies 
of  a  referenced  data  item  exists  in  the  DSH-1  storage  lev¬ 
els.  The  two-level  store-behind  operation  also  maintains  at 
least  two  copies  of  any  updated  data  item  in  DSH-1.  This  is 
a  key  reliability  feature  of  DSH-1.  It  is  important  to  know 
under  what  conditions  and  using  what  types  of  algorithms  can 
this  multiple  data  redundancy  be  maintained  at  all  times. 

3.4.3  Automatic  Data  Repair  Algorithms 

One  of  the  benefits  of  maintaining  redundant  data  in 
DSH-1  is  that  lost  data  due  to  component  failures  can  be 
reconstructed  on  a  spare  component  from  a  copy  of  the  lost 
data.  By  using  automatic  data  repair  in  DSH-1  the  probabil¬ 
ity  of  multiple  data  loss  can  be  reduced. 
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Two  classes  of  automatic  data  repair  algorithms  are  pos¬ 
sible.  One  strategy  is  to  make  use  of  the  multiple  data 
redundancy  properties  of  DSH-1  and  to  reconstruct  the  lost 
data  from  its  copy  in  a  different  storage  level.  The  other 
approach  is  to  maintain  duplicate  copies  of  the  data  item 
within  a  storage  level  and  to  reconstruct  the  lost  data  from 
its  copy  in  the  same  storage  level.  The  latter  approach  is 
particularly  attractive  for  low  performance  devices  such  as 
mass  storage. 

3.4.4  Performance  Evaluation 

A  key  issue  in  the  DSH-1  design  is  predicting  its  perfor¬ 
mance.  In  order  to  accomplish  this,  a  simplified  design  of 
DSH-1  and  its  algorithms  can  be  developed.  A  simulation 
model  can  then  be  developed  for  this  design.  Various  basic 
performance  statistics  can  then  be  obtained  under  various 
load  assumptions.  This  experiment  will  provide  insights  and 
directions  for  further  design  efforts. 

3. 5  SUMMARY 

The  INFOPLEX  storage  hierarchy  is  a  high  performance  high 
availability  virtual  memory  data  storage  hierarchy  with  dis¬ 
tributed  controls  for  data  movement  and  address  translation. 
It  is  designed  specifically  to  provide  a  very  large  perman¬ 
ent  virtual  address  space  to  support  multiple  functional 
hierarchy  processors. 
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A  general  structure  of  DSH-1,  the  INFOPLEX  storage  hier¬ 
archy  has  been  described  in  this  chapter.  This  general 
structure  can  be  used  to  derive  a  large  number  of  alterna¬ 
tive  configurations  which  can  be  used  to  explore  various 
algorithms  for  data  storage  hierarchy  systems. 

A  number  of  important  design  issues  associated  with  DSH-1 
are  also  outlined. 
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Chapter  IV 

MODELLING  AND  ANALYSIS  OF  DATA  STORAGE  HIERARCHY  SYSTEMS 

4.1  INTRODUCTION 

This  chapter  is  aimed  at  modelling  data  storage  hierarchy 
systems  so  as  to  study  these  systems  from  a  theoretic  point 
of  view.  These  studies  provide  insights  to  the  perfromance 
and  reliability  properties  of  data  storage  hierarchy  systems 
and  their  algorithms.  These  insights  provide  guidance  to 
developing  effective  data  storage  hierarchy  systems. 

Current  research  in  storage  hierarchy  systems  is  reviewed 
and  extended.  A  formal  model  of  a  data  storage  hierarchy 
which  incorporates  multiple  page  sizes  and  maintains  multi¬ 
ple  data  redundancy  is  developed.  The  LRU  algorithm  is 
extended  to  include  the  read-through  and  overflow  handling 
strategies  in  a  multi-level  storage  hierarchy.  Formal 
definitions  for  these  extended  algorithms  are  developed. 
Finally,  important  performance  and  reliability  properties  of 
data  storage  hierarchy  systems  are  identified  and  analyzed 
in  detail. 
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4.2  RESEARCH  ON  STORAGE  HIERARCHY  SYSTEMS 


Two  and  three-level  memory  hierarchies  have  been  used  in 
practical  computer  systems  (Conti,  1969;  Johnson,  1975; 
Greeberg  and  Webber,  1975).  However,  there  is  relatively 
little  experience  with  general  hierarchical  storage  systems. 

One  major  area  of  theoretic  study  of  storage  hierarchy 
systems  in  the  past  has  been  the  optimal  placement  of  infor¬ 
mation  in  a  storage  hierarchy  system.  Three  approaches  to 
this  problem  have  been  used;  (1)  Static  placement  (Rama- 
moorthy  and  Chandy,  1970;  Arola  and  Gallo,  1971;  Chen,  1973) 
-  this  approach  determines  the  optimal  placement  strategy 
statically,  at  the  initiation  of  the  system;  (2)  Dynamic 
placement  (Lum  et  1975;  Franaszek  and  Bennett,  1978)  - 
this  approach  attempts  to  optimally  place  information  in  the 
hierarchy,  taking  into  account  the  dynamically  changing 
nature  of  access  to  information;  (3)  Information  structur¬ 
ing  (Hatfield  and  Gerald,  1971;  Johnson  J.,  1975)  -  this 
approach  manipulates  the  internal  structure  of  information 
so  that  information  items  that  are  frequently  used  together 
are  placed  adjacent  to  each  other. 

Another  major  area  of  theoretic  study  of  storage  hier¬ 
archy  systems  has  been  the  study  of  storage  management 
algorithms  (Belady,  1966;  Belady  et  al,  1969;  Denning,  1970; 

t 

Mattson  e;t  1970;  Mattson,  1971;  Hatfield,  1972;  Madnick, 
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1973;  Goldberg,  1974;  Franklin  1978).  Here  the  study 

of  storage  hierarchies  and  the  study  of  virtual  memory  sys¬ 
tems  for  program  storage  have  overlapped  considerably.  This 
is  largely  due  the  fact  that  most  of  the  studies  of  storage 
hierarchies  in  the  past  have  been  aimed  at  providing  a  vir¬ 
tual  memory  for  program  storage.  These  studies  usually  do 
not  consider  the  effects  of  multiple  page  sizes  across  sto¬ 
rage  levels,  nor  the  problem  of  providing  redundant  data 
across  storage  levels  as  used  in  the  system  proposed  by  Mad- 
nick  (Madnick,  1973).  These  considerations  are  of  great 
importance  for  a  storage  hierarchy  designed  specifically  for 
very  large  data  bases.  The  following  sections  extend  theo¬ 
ries  on  storage  hierarchy  to  include  systems  that  incorpo¬ 
rate  multiple  page  sizes  and  maintains  multiple  data  redun¬ 
dancy. 
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4 . 3  MODEL  OF  A  DATA  STORAGE  HIERARCHY 


A  data  storage  hierarchy  consists  of  h  levels  of  storage 
1  2  h 

devices,  M  ,  M  ,  M  .  The  page  size  of  is  and 

the  size  of  is  pages  each  of  size  is  always 

an  integral  multiple  of  for  i=2,3 _ _  h.  The  unit 

of  information  transfer  between  and  is  a  page,  of 

size  Q^.  Figure  4.1  illustrates  this  model  of  the  data 
storage  hierarchy. 

All  references  are  directed  to  M^.  The  storage  manage¬ 
ment  algorithms  automatically  transfer  information  among 
storage  levels.  As  a  result,  the  data  storage  hierarchy 
appears  to  the  reference  source  as  a  storage  device 
with  the  size  of  M  . 

As  a  result  of  the  storage  management  algorithms  (to 
be  discussed  next) ,  multiple  copies  of  the  same  infor¬ 
mation  may  exist  in  different  storage  levels. 
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Figure  4.1  Model  of  a  Data  Storage  Hierarchy 


4.3.1  Storage  Management  Algorithms 

We  shall  focus  our  attentions  on  the  basic  algorithms 
to  support  the  read-through  (Madnick,  1975)  operation. 
Algorithms  to  support  other  operations  can  be  derived  from 
these  basic  algorithms. 

In  a  read-through,  the  highest  storage  level  that  con¬ 
tains  the  addressed  information  broadcasts  the  information 
to  all  upper  storage  levels,  each  of  which  simultaneously 
extracts  the  page  (of  the  appropriate  size)  that  contains 
the  information  from  the  broadcast.  If  the  addressed  in¬ 


formation  is  found  in  the  highest  storage  level,  the 
read-through  reduces  to  a  simple  reference  to  the  address¬ 
ed  information  in  that  level.  Figure  4.2  illustrates  the 
read-through  operation. 

Note  that  in  order  to  load  a  new  page  into  a  storage 
level  an  existing  page  may  have  to  be  displaced  from  that 
storage  level.  We  refer  to  this  phenomenon  as  overflow. 
Hence,  the  basic  reference  cycle  consists  of  two  sub-cycles, 
the  read-through  cycle  (RT) ,  and  the  overflow  handling 
cycle  (OH) ,  with  RT  preceeding  OH. 


For  example.  Figure  4.2  illustrates  the  basic  refer¬ 


ence  cycle  to  handle  a  reference  to  the  page  p  .  During 

ya 

the  Read-Through  (RT)  subcycle,  the  highest  storage  level 

X  1  *1 

(M  )  that  contains  p^^  broadcasts  the  page  containing 

to  all  upper  storage  levels,  each  of  which  extracts  the 
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Figure  4 . 2  The  Read  Through  Operation 
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page  of  appropriate  size  that  contains  from  the  broadcast. 

ya 

As  result  of  the  Read-Through,  there  may  be  overflow  from 
the  storage  levels.  These  are  handled  in  the  Overflow- 
Handling  (OH)  subcycle. 

It  is  necessary  to  consider  overflow  handling  because 
it  is  desirable  to  have  information  overflowed  from  a 
storage  level  to  be  in  the  immediate  lower  storage  level, 
which  can  then  be  viewed  as  an  extension  to  the  higher 
storage  level. 

One  strategy  of  handling  overflow  to  meet  this  objec¬ 
tive  is  to  treat  overflows  from  as  references  to 
We  refer  to  algorithms  that  incorporate  this  strategy  as 
having  dynamic-overflow-placement  (OOP) . 

Another  possible  overflow  handling  strategy  is  to 

i 

treat  an  overflow  from  M  as  a  reference  to  M  only 
when  the  overflow  information  is  not  already  in  If 

the  overflow  information  is  already  in  no  overflow 

handling  is  necessary.  We  refer  to  algorithms  that  in¬ 
corporate  this  strategy  as  having  static-overflow-place¬ 
ment  (SOP) . 

Let  us  consider  the  algorithms  at  each  storage  level 
for  selecting  the  page  to  be  overflowed.  Since  the  Least 
Recently  Used  (LRU)  algorithm  (Denning,  1974;  Mattson  et . 
al,  1970)  serves  as  the  basis  for  most  current  algorithms, 
we  shall  consider  natural  extensions  to  LRU  for  managing 
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the  storage  levels  in  the  data  storage  hierarchy  system. 

Consider  the  following  two  strategies  for  handling  the 
Read-Through  Cycle.  First,  let  every  storage  level  above 
and  including  the  level  containing  the  addressed  infor¬ 
mation  be  updated  according  to  the  LRU  strategy.  Thus, 
all  storage  levels  lower  than  the  addressed  information 
do  not  know  about  the  reference.  This  class  of  algorithms 
is  called  LOCAL- LRU  algorithm.  This  is  illustrated  in 
Figure  4.3. 

The  other  class  of  algorithms  that  we  shall  consider 
is  called  GLOBAL-LRU  algorithm.  In  this  case,  all  storage 
levels  are  updated  according  to  the  LRU  strategy  whether 
or  not  that  level  actually  participates  in  the  read- 
through.  This  is  illustrated  in  Figure  4.4. 

Although  the  read-through  operation  leaves  supersets 

of  the  page  p^  in  all  levels,  the  future  handling  of 
ya 

each  of  these  pages  depends  upon  the  replacement  algo¬ 
rithms  used  and  the  effects  of  the  overflow  handling.  We 
would  like  to  guarantee  that  the  contents  of  each  storage 
level,  M^,  is  always  a  superset  of  its  immediately  higher 
level,  This  property  is  called  Multi-Level  Inclu¬ 

sion  (MLI) .  Conditions  to  guarantee  MLI  will  be  derived 
in  a  later  section. 

It  is  not  difficult  to  demonstrate  situations  where 
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Figure  4.3  Local-LRU  Algorithm 
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Reference  to  P 


is  the  highest 
level  where  P  is  found 


Figure  4.4  Global-LRU  Algorithm 


handling  overflows  generates  references  which  produce 
overflows,  which  generate  yet  more  references.  Hence 
another  important  question  to  resolve  is  to  determine 
the  conditions  under  which  an  overflow  from  is  always 
found  to  already  exist  in  i.e.,  no  reference  to 

storage  levels  lower  than  is  generated  as  a  result 

of  the  overflow.  This  property  is  called  Multi-Level 
Overflow  Inclusion  (MLOI) .  Conditions  to  guarantee  MLOI 
will  be  derived  in  a  later  section. 

We  shall  consider  these  important  properties  in  light 
of  four  basic  algorithm  alternatives  based  on  local  or 
global  LRU  and  static  or  dynamic  overflow.  Formal 
definitions  for  these  algorithms  will  be  provided  after 
the  basic  model  of  the  data  storage  hierarchy  system  is 
introduced. 
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4.3.2  Basic  Model  of  a  Data  Storage  Hierarchy 

For  the  purposes  of  this  thesis,  the  basic  model  illus¬ 
trated  in  Figure  4.5  is  sufficient  to  model  a  data  storage 
hierarchy.  As  far  as  the  Read-Through  and  Overflow- 
Handling  operations  are  concerned,  this  basic  model  is 
generalizable  to  a  h-level  storage  hierarchy  system. 

can  be  viewed  as  a  reservoir  which  contains  all  the 
information.  is  the  top  level.  It  has  m^  pages  each 

of  size  (j=i+l)  is  the  next  level.  It  has  m^ 

pages  each  of  size  nQ^^  where  n  is  an  integer  greater  than 

1. 
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4.3.3  Formal  Definitions  of  Storage  Management  Algorithms 


Denote  a  reference  string  by  r  =  r^,  ...,  ' 

where  r^  is  page  being  referenced  at  the  t-th 

reference  cycle.  Let  be  the  stack  for  at  the  begin¬ 
ning  of  the  t-th  reference  cycle,  ordered  according  to  LRU. 
That  is,  sj  =  (S^(l),  sj(2),  ...,  S^(K)),  where  S^{1)  is  the 
most  recently  referenced  page  and  S^(K)  is  the  least 
recently  referenced  page.  Note  that  =  capacity 

of  in  terms  of  the  number  of  pages) ,  The  number  of 
pages  in  is  denoted  as  |S^1  ,  hence  |S^|  =  K.  By  con¬ 
vention,  =  J2f,  js^l  =0. 

i  i  i 

S.  IS  an  ordered  set.  Define  M.  as  the  contents  of  S. 
t  t  t 

without  any  ordering.  Similarly,  we  can  define  and 
for  . 


Let  us  denote  the  pages  in  by  P^,  P^, 


Each 


page,  P^,  in  M-* ,  consists  of  an  equivalent  of  n  smaller 
pages,  each  of  size  =  Q./n.  Denote  this  set  of  pages 


by  (pj)S  i.e.,  (pj)i  ={l=Yl' 


In  general. 


(M^)  ^  is  the  set  of  pages,  each  of  size  obtained  by 

"breaking  down"  the  pages  in  M^.  Formally,  ^  = 

(S^(k))^  where  x  =  | |  .  is  called  the 

1—  1  ^  ^  y 


family  from  the  parent 


Py.  Any  pair  of  pages,  P^^ 


and  Pyj^  from  (P^)^  are  said  to  be  family  equivalent, 
denoted  by  P^^  =  Py^*  Furthermore,  a  parent  page  P^  and 
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a  page  (for  l^z^n)  from  its  family  are  said  to  be 

i  c  i 

corresponding  pages,  denoted  by 

and  are  said  to  be  in  corresponding  order,  de¬ 
noted  by  =  S^,  if  S^(k)  =  S^(k)  for  k  =  1,  2,  3,  . . .w, 
where  w  =  min  (|s^|,  |S^|  ).  Intuitively,  two  stacks  are 
in  corresponding  order  if,  for  each  element  of  the  shorter 
stack,  there  is  a  corresponding  page  in  the  other  stack 
at  the  same  stack  distance  (the  stack  distance  for  page 
S^(k)  is  defined  to  be  k) . 

and  are  said  to  be  correspondingly  equivalent, 
denoted  by  if  =|M^|  and  for  any  k  =  1,  2, 

there  exists  x,  such  that  S^(k)  = 

S^(x)  P  (y)  for  all  y  k.  Intuitively,  the  two  memo¬ 
ries  are  correspondingly  equivalent  when  each  page  in 
one  memory  corresponds  to  exactly  one  page  in  the  other 
memory . 

_ 2_  i 

The  reduced  stack,  S^,  of  is  defined  to  be  S^(k)  = 
S^(jk)  for  k  =  1,  ...,  Is^l  where  is  the  minimum 
where  ^  ^k' 

Intuitively,  is  obtained  from  by  collecting  one 
page  from  each  family  existing  in  S^,  such  that  the  page 
being  collected  from  each  family  is  the  page  that  has  the 
smallest  stack  distance  within  the  family. 

In  the  following,  we  define  the  storage  management 
algorithms.  In  each  case,  assume  that  the  page  referenced 
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at  time  t  is  P 


ya 


LRU  (sj,  )  =  si 
t'  ya  t 

P 


ya 
Case  1; 


Case  2 


ya 

.i 


is  defined  as  follows: 

i 
t 

(X)  =' 


t+1 

i  „i  „i 
t '  ya 


e  s:,  P^  =  si(k)  : 


=  Pya'  "i+1 


4^(x-l),  l<x^k 


^S^(x),  k<x^[S^| 


^  si  : 

ya  t 


t+1 


(1)  =  Si^,  (X)  =  si(x-l)  , 


ya'  ^t+1 

1  <  X  ^  min  (m^^ , 


s;i  +  1) 


If  Is  I  =  m.  then  P^  =  si (m. )  is  the  over- 

I-  1  Ocl  u  X 

flow,  else  there  is  no  overflow. 
LOCAL-LRU-SOP  (si,  S^ ,  pi,)  =  (si_,,  ,  S^J  is  defined 


^t'  “t'  ya' 

as  follows: 


t+1'  "t+1' 


Case  1:  P^  e  si  : 

-  ya  t 

sj^l  =  (sj,  pj^),  =  sj 

C.?g^2=  Pya  ®t'  = 

s^,  =  {sj,  pj^),  sj,  =  (sj,  pj). 

If  there  is  no  overflow  from  S^ 

then  S^_^^  =  S^,  and  S^^^  =  S^, 

If  overflow  from  si  is  the  page  P^ 

t  ^  oa 

then  =  SOP  (sj.,  sj,  ,  pj^) 

defined  as: 

^  ^i'  =t+i  = 

if  pj  ^  sj,  then  S^^^  =  LRU  (s2,,  P^) 
o  t'  t+1  -  t'  o 
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Case  3; 

LOCAL-LRU-DOP 
Case  1: 

Case  2: 


Case  3: 

GLOBAL-LRU-SOP 


P^  i  st  and  ja!  s2  : 
ya  t  y  t 


(handled  as  in  Case  2) 

(S^,  ,  P^  )  =  (S^ , , ,  )  is  defined  as 

t  t  ya  t+1  t+1 


P  e  S7 
ya  t 


sr.  ,  =  LRU  (Sj,  P,^_)  ,  Sj 


,1 

't+1 


ya 


t+1  ^t 


P^  /  and  P^  e  : 
ya  ^  t  y  t 

St-  =  Py^)  ,  sj,  =  ^  (S^,  Pp 

If  no  overflow  from  then  =  S^, 

t  t+i  t 


and  S 


t+1 


= 


t' 


If  overflow  from  is  P^  then 

t  oa 

^^t+1'  ^t+1^  "  —  ^^t’'  ^t*'  ^oa^  which  is 
defined  as; 

st+i=  s7  and  (sj,,  p3) 

P^  ^  and  P^  ^  : 

ya  t  y  t 

(handled  as  in  Case  2  above) 

(si,  sj,  P,^_)  =  is  defined  as 


t '  ya 

follows : 

„i 


LRU  (sS  P.^-_)  and  S?  ,  =  LRU  (S^,  Pp.)  , 

— t  ^  y 

i  . , 


t'  ^  ■  f  ya 

If  no  overflow  from  S'T  then  S 


t+1  ^t'  and 


't+1 


■■i- 


If  overflow  from  is  P^  then  (S^  , ,  Sp  ) 

t  oa  t+i  t+i 

=  SOP  (sj,,  sj,, 
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GLOBAL-LRU-DOP  (S^,  sj,  ^t+1^  defined  as 

S^,  =  (S^,  Py^)  and  sj,  =  ^  (sj,  P^) 

If  no  overflow  from  then  ,  and  , 

t  t+1  t'  t+1  t' 

If  overflow  from  is  P^  then  (S^, ^ . , )  = 

t  oa  t+1  t+1 

°°E  O 
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4.4  PROPERTIES  OF  DATA  STORAGE  HIERARCHY  SYSTEMS 


One  of  the  properties  of  a  Read-Through  operation  is 
that  it  leaves  a  "shadow"  of  the  referenced  page  (i.e., 
the  corresponding  pages)  in  all  storage  levels.  This  pro¬ 
vides  multiple  redundancy  for  the  page.  Does  this  multiple 
redundancy  exist  at  all  times?  That  is,  if  a  page  exists 
in  storage  level  ,  will  its  corresponding  pages  always 
be  in  all  storage  levels  lower  than  M^?  We  refer  to  this 
as  the  Multi-Level  Inclusion  (MLI)  property.  As  illus¬ 
trated  in  Figure  4.6  for  the  LOCAL-LRU  algorithms  and  in 
Figure  4.7  for  the  GLOBAL-LRU  algorithms,  it  is  not 
always  possible  to  guarantee  that  the  MLI  property  holds. 
For  example,  after  the  reference  to  P^j^  in  Figure  4.6(a) 
the  page  P^^^  exists  in  M^  but  its  corresponding  page  P^ 
is  not  found  in  M^ .  In  this  chapter  we  shall  derive  the 
necessary  and  sufficient  conditions  for  the  MLI  property 
to  hold  at  all  times. 

Another  desirable  property  of  the  data  storage  hierarchy 
is  to  avoid  generating  references  due  to  overflows.  That 
is,  under  what  conditions  will  overflow  pages  from  M^ 
find  their  corresponding  pages  already  existing  in  the 
storage  level  M^^^?  We  refer  to  this  as  the  Multi-Level 
Overflow  Inclusion  (MLOI)  property.  We  shall  investigate 
the  conditions  that  make  this  property  true  at  all  times. 
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(a)  lcx:al-lru-sop 


Figure  4.6  Violation  of  MLI  by  Local-LRU  Algorithms 
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Figure  4.7 


(b)  GLOBAL-LRU-DOP 


Of  MLI  by  Global-LHU  Algorithms 


Referring  to  the  basic  model  of  a  data  storage  hierarchy 
in  Figure  4.5,  for  high  performance  it  is  desirable  to 
minimize  the  number  of  references  to  (the  reservoir) . 

If  we  increased  the  number  of  pages  in  ,  or  in  ,  or 
in  both,  we  might  expect  the  number  of  references  to  M 
to  decrease.  As  illustrated  in  Figure  4.8  for  the  LOCAL- 
LRU-SOP  algorithm,  this  is  not  always  so,  i.e.,  for  the 
same  reference  string,  the  nxamber  of  references  to  the 
reservoir  actually  increased  from  4  to  5  after  is 
increased  by  1  page  in  size.  We  refer  to  this  phenomena 
as  a  Multi-Level  Paging  Anomaly  (MLPA) .  One  can  easily 
find  situations  where  MLPA  occurs  for  the  other  three 
algorithms.  Since  occurrence  of  MLPA  reduces  performance 
in  spite  of  the  costs  of  increasing  memory  sizes,  we 
would  like  to  investigate  the  conditions  to  guarantee 
that  MLPA  does  not  exist. 
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reference  to  M* 

P  1 1  j  f’zi  1  P  ll  1  P31  '  P'li  ’  P41 

-  1  1  1  1  1 

contents  of  M* 

(mj  -  2) 

p'lp'lp'lp*  Ip*  Ip* 

'^II  1 1  ^11  r  31  1  II  |'4| 

'PiijPalPi'i  Ipj,  1p,‘i 

1  '  1  1  1 

overflow  from  M* 

i  i  ! 

reference  to  M* 

fl  ^  ifs  ^  i  *  Ip,'  # 

contents  of 

(mj  =  2) 

pj  {  p  J  j  pi  j  pi  pi  1  pi  1  pi  pi 

1  1^2  1  *^2  1  ^3  ^3  *3  ,  U 

|p/  [pj  IpI  p1  1p2  IP3'  P^ 

reference  to 

»  1  -ie  1  ^  ^  ^  1*  4, 

number  of  references  to  =  4 


reference  to  M  * 

p *  Ip'  1  p'  IP*  Ip*  Ip* 

^11  ,  ^21  ni  if^3i|  Hi  P41 

_ _  1  ' 

contents  of  M* 

(mj  =  3) 

1 

overflow  from  M 

!  '  i  i  1  ^1 

reference  to 

pi  i  pi  1  .  1  pi  *  1  I  pi  pi 

^1  1  ^2  ,  1  P3  1  P4  ^2 

contents  of  M  * 

(mp  2) 

pi  ipi  ipi  ipi  ipi  Ipi  pj 

1  1  2  ,  2  ,  3  1  3  1  4  2 

|p,'  .|pi  jp!  |pi  jpi  pi 

reference  to 

*  *l'#>|-*|^.  1*  -if 

- 1 _ 1 _ 1_^  1  1 

j  number  of  references  to  =  5 

Figure  4,8  Illustration  of  MLPA 
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4.4.1  Summary  of  Properties 


The  MLI,  MLOI,  and  MLPA  properties  of  the  data 
storage  hierarchy  have  been  derived  in  the  form  of  eight 
theorems.  These  theorems  are  briefly  explained  and 
summarized  below  and  formally  proven  in  the  following 
section. 

Multi-Level  Inclusion  (MLI)  :  It  is  shown  in  Theorem 
1  that  if  the  number  of  pages  in  is  greater  than  the 
number  of  pages  in  (note  pages  are  larger  than 
those  of  M^) ,  then  it  is  not  possible  to  guarantee  MLI 
for  all  reference  strings  at  all  times.  It  turns  out 
that  using  LOCAL-LRU-SOP ,  or  LOCAL-LRU-DOP ,  no  matter  how 
many  pages  are  in  M^  or  M^,  one  can  always  find  a  refer¬ 
ence  string  that  violates  the  MLI  property  (Theorem  2) . 
Using  the  GLOBAL-LRU  algorithms,  however,  conditions  to 
guarantee  MLI  exist.  For  the  GLOBAL-LRU-SOP  algorithm,  a 
necessary  and  sufficient  condition  to  guarantee  that  MLI 
holds  at  all  times  for  any  reference  string  is  that  the 
number  of  pages  in  M^  be  greater  than  the  number  of  pages 
in  M^  (Theorem  3).  For  the  GLOBAL-LRU-DOP  algorithm,  a 
necessary  and  sufficient  condition  to  guarantee  MLI  is 
that  the  ntimber  of  pages  in  M^  be  greater  than  or  equal 
to  twice  the  number  of  pages  in  M^  (Theorem  4) . 

Multi-Level  Overflow  Inclusion  (MLOI)  ;  It  is  obvious 
that  if  MLI  cannot  be  guaranteed  then  MLOI  cannot  be 
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guaranteed.  Thus,  the  LOCAL-LRU  algorithms  cannot  guaran¬ 
tee  MLOI.  For  the  GLOBAL-LRU-SOP  algorithm,  a  necessary 
and  sufficient  condition  to  guarantee  MLOI  is  the  same 
condition  as  that  to  guarantee  MLI  (Theorem  5) .  For  the 
GLOBAL-LRU-DOP  algorithm,  a  necessary  and  sufficient 
condition  to  guarantee  MLOI  is  that  the  number  of  pages 
in  M^  is  strictly  greater  than  twice  the  number  of  pages 
in  M^  (Theorem  6).  Thus,  for  the  GLOBAL-LRU-DOP  algo¬ 
rithm,  guaranteeing  that  MLOI  holds  will  also  guarantee 
that  MLI  will  hold,  but  not  vice  versa. 

Multi-Level  Paging  Anomaly  (MLPA)  :  We  have  identified 
and  proved  sufficiency  conditions  to  avoid  MLPA  for  the 
GLOBAL-LRU  algorithms.  For  the  GLOBAL-LRU-SOP  algorithm, 
this  condition  is  that  the  number  of  pages  in  M^  must  be 
greater  than  the  number  of  pages  in  M^  before  and  after 
any  increase  in  the  sizes  of  the  levels  (Theorem  7) .  For 
the  GLOBAL-LRU-DOP  algorithm,  this  condition  is  that 
the  number  of  pages  in  M^  must  be  greater  than  twice  the 
number  of  pages  in  M^  before  and  after  any  increase  in 
the  sizes  of  the  levels  (Theorem  8) . 

In  summary,  we  have  shown  that  for  the  LOCAL-LRU  algo¬ 
rithms,  no  choice  of  sizes  for  the  storage  levels  can 
guarantee  that  a  lower  storage  level  always  contains  all 
the  information  in  the  higher  storage  levels.  For  the 
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GLOBAL-LRU  algorithms,  by  choosing  appropriate  sizes  for 
the  storage  levels,  we  can  (1)  ensure  that  the  above 
inclusion  property  holds  at  all  times  for  all  reference 
strings,  (2)  guarantee  that  no  extra  page  references  to 
lower  storage  levels  are  generated  as  a  result  of  handling 
overflows,  and  (3)  guarantee  that  increasing  the  sizes 
of  the  storage  levels  does  not  increase  the  number  of 
references  to  lower  storage  levels.  These  results  are 
formally  stated  as  the  following  eight  Theorems.  Formal 
proofs  of  these  Theorems  are  presented  in  the  following 
section. 
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THEOREM  1 


Under  LOCAL-LRU-SOP ,  or  LOCAL-LRU-DOP ,  or  GLOBAL- 

LRU-SOP,  or  GLOBAL-LRU-DOP ,  for  any  m.>2,  m.<m. 

1—  1 


implies 
THEOREM  2 


3  r,t, 


m: 


Under  LOCAL-LRU-SOP,  or  LOCAL-LRU-DOP,  for  any 
m^^2,  and  any  m .  ,  ^r,t,  (M^)  ^  ;^M^ 


THEOREM  3 


Under  GLOBAL-LRU-SOP ,  for  any  .¥  r,t, 


M.  iff  m.  >  m. 
t  J  1 


THEOREM  4 


Under  GLOBAL-LRU-DOP,  for  any  m^^2  ,  ■^/"r ,  t ,  (M;^) 

M^  iff  m.  >  2m. 
t  3-1 


3n1- 


THEOREM  5 


Under  GLOBAL-LRU-SOP,  for  any  m^  ^  2,"V^,t, 
flow  from  M^  finds  its  corresponding  page  in  M^ 


an  over- 


iff  m.  >  m. 
3  1 


THEOREM  6 


an 


Under  GLOBAL-LRU-DOP ,  for  any  m^  ^  2 ,  Yr.t, 

overflow  from  M^  finds  its  corresponding  page  in 

M^  iff  m.  >  2m. 

3  1 


THEOREM  7 


Let  M^  {with  m^  pages) ,  M^  (  with  m^  pages)  and  M^ 
be  System  A. 
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Let  M'^  (with  pages),  M'^  (with  '  pages)  and 
M  be  System  B. 

Let  m. '  >  m.  and  m.'  >  m  .  Under  GLOBAL- LRU-SOP , 
1-1  3  -  j 

for  any  ^  2 ,  no  MLPA  can  exist  if 
m^  >  m^  and  m ^ ' 

THEOREM  8 

Let  System  A  and  System  B  be  defined  as  in  Theorem  7. 
Let  m^^ '  ^  j  '  j  •  Under  GLOBAL-LRU-DOP , 

for  any  m^  ^  2,  no  MLPA  can  exist  if  m^  > 

2m^  and  m ^ '  >  2m^ ' 
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4.4.2  Derivation  of  Properties 


THEOREM  1 


Under  LOCAL-LRU-SOP ,  or  LOCAL-LRU-DOP ,  or  GLOBAL- 
LRU-SOP ,  or  GLOBAL-LRU-DOP ,  for  any  m .  >  2 ,  m .  <  m . 
implies  r,t, 


PROOF 


Case  1  :  m^  < 

Consider  the  reference  string  r=”  P^  ,  P?;  ,  ... 

^  la'  2a 

P^  " 

""(m.+Da  • 

Using  any  one  of  the  algorithms,  the  following 
stacks  are  obtained  at  t=mj+2  : 

^t  =  4a . 4^ 


- 


=  (P 


(mj+1) 


I  f  •  •  •  f  P-} »  P-i^ 


m 


Thus,  P^g^  e  but  P^^  ^  i.e.,  (M^)  ^  ^ 

Case  2 


M 


_  m^  =  m^  =  w 

Consider  the  reference  string  r  =  "  P^^»  ^2a'  ***' 

P^  " 

^(w+l)a 

Using  any  one  of  the  above  algorithms ,  the  fo] lowing 
stacks  are  obtained  at  t=w+2  : 

<=  ia' 

sj  =  (p’,  P3) 


Thus,  P2^  e  but  P^^  i.e.,  ^ 


'M 


Q.E.D. 
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rt  H' 


THEOREM  2 


Under  LOCAL-LRU- SOP^  or  LOCAL- LRU-DOP ,  for  any 
^  2,  and  any 

PROOF  (For  LOCAL-LRU-SOP) 

For  the  result  follows  directly  from  THEOREM  1 


For  using  the  reference  string 

za'  ia-  za'  2a'  •**'  za'  m^a  ' 
the  following  stacks  will  be  produced  at  t=2mj+l  : 


T-  =  "  p^  P^  P^ 

'  la'  '^za’ 


=  ''“m.a' 


P^  P^ 
za'  (mj-l)a' 


(m  .-ra.  +2)  a 
3  1 


si  =  (P;,  , 


m , 


pi 

m.-l' 

3 


pi  p^  \ 
•  ••/  ■*^2'  ’*^1' 


Thus,  P^^  e  IaI  but  P^^  ^  (M^)^,  i.e.,  (M^)^j;6m?;. 

Za  u  Za  u  u  ““  u 

Q. E .D . 

PROOF  (For  LOCAL-LRU-DOP) 


For  the  result  follows  directly  from  THEOREM  1 

For  mj>mj^,  using  the  following  reference  string 


r  =  "  p^  ,  P^  ,  P^  ,  P^  ,  . 

za  la  za  2a 


P^  ,  P^  ", 
za  m.a 
3 


The  following  stacks  will  be  produced  at  t=2mj+l  ; 


=  (p!: 


m.a'  za 
3 

sj  =  (a^,  a2,  .. 


Where  for  l<i<m. 

- 3 


P  ) 

'  (m.-m.+2)  a'  ' 

3  1 


m . 
3 


'  tv  "Vi . "5'  "2'  "p’ 


since  P^  is  the  only  overflow  from  M^ . 
z 

Thus,  P^  e  Mf'  but  P^  ^  (M^)^ 
za  t  za  t 


,  i.e. ,  (m2) M^. 

t  —  t 

Q.E.D. 
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THEOREM  3 


Under  GLOBAL-LRU-SOP ,  for  any  >  2 ,  V  r,t,  (mJ) ^  2 

iff  m.>m. 
t  j  1 


PROOF 

This  proof  has  two  parts.  Part  (a)  to  prove  V  r,t, 

{Mi[)  =>  m.  >  iti. 

t  —  t  3  1 

or  equivalently,  ra . <  m .  ^ 3 r , t , (M? ) 

3  —  1  t  +■  t 

Part  (b)  to  prove  Vr,  t,  (M^) 

PROOF  of  Part  (a)  ;  r ,  t,  (M^) 

This  follows  directly  from  THEOREM  1. 

Q.E.D. 

To  Prove  Part  (b) ,  we  need  the  following  results. 

LEMMA  3.1 

V  r,t  such  that  Im^I  <m.,  ifm.  =m.  +  1,  then 

'  t '  —  1  3  1 

(a)  (Mj)^2M^,  and  (b)  sj  =  sj 

PROOF  of  LEMMA  3.1 

For  t=2  (i.e.,  after  the  first  reference),  (a)  and  (b) 
are  true.  Suppose  (a)  and  (b)  are  true  for  t,  such 
that  Im^ 1 <  m. 

Consider  the  next  reference: 

Case  li  It  is  a  reference  to  M^: 

There  is  no  overflow  from  M^  or  M^ ,  so  (a)  is  still 
true.  Since  Global-LRU  is  used,  (b)  is  still  true. 
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Case  2;  It  is  a  reference  to  : 

There  is  no  overflow  from  .  If  no  overflow  from 
M^,  the  same  arguement  as  Case  1  applies.  If  there  is 
overflow  from  M^,  the  overflow  page  finds  its  corres¬ 
ponding  page  in  .  Since  SOP  is  used,  this  overflow 
can  be  treated  as  a  "no-op" .  Thus  (a)  and  (b)  are 
preserved. 

Case  3 ;  It  is  a  reference  to  M  ; 

There  is  no  overflow  from  since  |M2,,l<m.  .  Thus 

'  t+1'—  1 

the  same  reasoning  as  in  Case  2  applies. 

Q.E.D. 


LEMMA  3.2 

V  r,t,  such  that  |m^|  =  m . ,  if  m.^m.+l  then 

(a)  (Mj)^rjMj,  (b)  S^  ^  S^,  and  (c)  (S  j (m^ )  )  ^  fl  sJ=j2l 
Let  us  denote  the  conditions  (a)  (b)  and  (c)  jointly 

as  Z (t) . 

PROOF  of  LEMMA  3.2 

Suppose  the  first  time  S^Cm^)  is  filled  is  by  the 

t*-th  reference.  That  is,  S^(m. )  =  0  for  all  t_<t* 

and  sZ(m.)  0  for  all  t>t*.  From  LEMMA  3.1  we  know 
t  j 

that  (a)  and  (b)  are  true  for  all  t^t*. 

Let  tj^=t*  +  1,  =  t*  +  2,  ...,  etc.  We  shall  show, 

by  induction  on  t,  starting  '^at  t^,  that  A(t)  is  true. 
First  we  show  that  Z(t^)  is  true  as  follows; 
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Case  1 :  g 

si*  0  si*  and  M^*  e  Mi* 4 si*  (m . -1)  c  si*{in.) 
t*  =  t*  t*  =  t*  t*  j  =  t*  1 

As  a  result  of  the  reference  at  t*  (to  M  ) , 

si*.i(ni-)  =  si*(m.-l)  and  si*(in.)  overflows  from 
f'+i  j  t*  3  t*  1 

This  overflow  page  finds  its  corresponding  apge  in 
because  there  is  no  overflow  from  and  (a) .  Since 
SOP  is  used,  the  overflow  from  can  be  treated  as 
a  "no-op".  Furthermore,  since  Global-LRU  is  used, 

(b)  is  true  after  the  t*-th  reference,  (b)  and 
I  S^^_i_lj  >ls^*_i_l  I  =^(a)  and  (c)  .  Thus  Z(tj^)  is  true. 


Case  2:(M^*)^3M^*  and  and  ^  M^* 

(M^*)  and  ^  /3s^*(k)  such  that 

(S^*(k))^n  =  0  Sj:*  g  S^*  and  (S^*  (k)  )  mJ*  =  0 

k>|s^*|and  (S^*(x})^  =  0  for  all  x  where 

X  ^k.  Thus  (S^*  (mj_^)  )  ^  S^*  =  0  (i.e.,  the  last 

page  of  sj*  is  not  S^*) 

si*{m.)  overflows  from  M^.  There  is  no  overflow  from 
t*  1 

.  Thus  the  overflow  page  from  finds  its  corres¬ 
ponding  page  in  .  For  the  same  reasons  as  in  Case 
1,  (b)  is  still  preserved.  (b)  and 


(a)  and  (c)  are  true.  Thus,  Z(t^)  is  true. 

Ass\ime  that  Z(tj^)  is  true;  to  show  that  Z (t^^l)  is  true. 


we  consider  the  next  reference,  at  time  t 


k+l‘ 
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Imagine  that  the  last  page  of  does  not  exist, 

i.e.,  (m^)  =  0.  If  the  reference  at  is  to 


a  page  in  or  ,  then  (a)  and  (b)  still  hold 

k 

because  Global-LRU  is  used  and  because  overflow  from 
finds  its  corresponding  page  in  (See  the  proof 
of  LEMMA  3.1) . 

If  the  reference  at  is  to  a  page  not  in  M^  ,  then 

k 

we  can  apply  the  agruement  as  that  used  in  considering 
the  reference  at  time  t^  above  to  show  that 
still  true. 


Q.E.D. 


LEMMA  3.3 

V  r,t,  if  mj=m^+l  then  (a) 


PROOF  of  LEMMA  3.3 


and 


(b) 


<4'V>'n 


For  t  such  that  tM^|<m^(a)  follows  directly  from 

LEMMA  3.1  and  (b)  is  true  because  S^(m. )  =  0 

For  t  such  that  |M^|  =  m.(a)  and  (b)  follows  directly 

^  3 

from  LEMMA  3.2 


Q.E.D. 


LEMMA  3.4 

V  r,t,  if  m.>  m.  than  (a) 

(si(m.))^r\S^  =  0 
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PROOF  of  LEMMA  3.4 


Let  nij  =  m^+k.  We  shall  prove  this  lemma  by  induction 
of  k.  For  k=l  (a)  and  (b)  are  true  from  LEMMA  3.3. 

Suppose  that  (a)  and  (b)  are  true  for  k.  Consider 
m.=m.+(k+l).  That  is  consider  the  effects  of 
increasing  by  1  page  in  size: 

Since  M^  is  unchanged,  M^  (with  m^+k+1  pages)  sees 
the  same  reference  string  as  M^  (with  pages)  . 

Applying  the  stack  inclusion  property  (Mattson  et  al., 
1970)  ,  we  have  M^  (with  m^+k+1  pages)  2  M^  (with  m^^+k  pages)  . 
Thus  (a)  is  still  true.  Suppose  (S^  (m^+k+1)  )  ^(^  S^  0 
then  there  is  a  page  in  M^  that  corresponds  to  this 
page.  But  S^(m.+k+l)  is  not  in  M^  (with  m.+k  pages). 

This  contradicts  the  property  that  (M^)^oM^.  This 
showes  that  (b)  is  still  true. 

Q.E.D. 

PROOF  of  Part  (b)  ;  m.  >m.;:^V  r,t,  (Mj)^3Mj:; 

This  follows  directly  from  LEMMA  3.4. 

Q.E.D. 
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THEOREM  4 


Under  GLOBAL-LRU-DOP ,  for  any  2,  V  r,t,(M^)^  2. 

iff  m.>2m. 
t  D-  1 

PROOF 

This  proof  has  two  parts: 

Part  (a);  m  .  <2ia.  3  r ,  t, 

31  t  t 

Part  (b)  :  in.>2in.=&  V  r,t, 

3—  1  ^  '  '  t  ~  t 


PROOF  of  Part  (a):  m  .  <  2mj^ g  r ,  t ,  (M^)  ^  ^ 


For  the  result  follows  from  THEOREM  1. 

Consider  the  case  for  2m.  >m.  >m.; 

The  reference  string  r  =  "pj^^  ,  P^^,  P^^,  •  •  • /P^(2m .  )  a" 
will  produce  the  following  stacks: 

^  ^  (2m^-l)a'  ^  (m^+l)a^' 

=  (  a^,  a2,  ^  where  a.'s  are  picked  from 

and  L2  alternatively,  starting  from  L^. 


^1  ^^m.'  ^^(m.-l)'  •••'  ^1^ 


1  '  m.  '  "  (m.-l)  '  *  •  •  '  "1' 

X  1 

L  =r  fp3  p3  pD 

^2  '2m.'  (2m. -D'  *•*'  m.+l). 

11  1 

If  m.  is  even,  then  (a^,  a^,  ...  a^  corresponds  to 

the  frist  mj/2  elements  of  and  (a2,  a^,  ...  a^  ) 

corresponds  to  the  first  m./2  elements  in  L„.  We  see 

3  ^ 

that  is  in  but  its  corresponding  page 

is  not  in  i  \  is  not  in  since  m./2  <m^). 

t  (m^+1)  t  3'  i' 

If  m.  is  odd,  then  (a^,  a^,  ...  a^  )  corresponds  to  the 

3  nij 

first  {m.+l)/2  elements  in  L,  and  (a.,,  2.,  ...,  a  ,) 

3  1  2  4m .  -1 

3 
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corresponds  to  the  first  (m.-l)/2  elements  in  L^.  We 
see  that  the  page  +l)a  ^t  correspond 

ing  page  is  not  in  because  max (  (m.-l)/2)  =  m.-l, 
thus,  a,  is  at  most  the  (nu-D-th  element  of 

.  j 

'  ^2'  ^L.-(m.-l)+l=Pm.+2.  ^  both  cases, 

Q.E.D. 

To  prove  Part  (b) ,  we  need  the  following  preliminary 
results. 

LEMMA  4.1 

Under  GLOBAL-LRU-DOP ,  for  m.  >2,  m.  >2m.,  a  page 

1  —  3—1 

found  at  stack  distance  k  in  M^  implies  its  corres¬ 
ponding  page  can  be  found  within  stack  distance  2k  in 


PROOF  of  LEMMA  4.1 

We  prove  by  induction  on  t. 

At  t=l,  the  statement  is  trivially  true.  At  t=2 
(i.e.,  after  the  first  reference)  S^(l)  and  its 
corresponding  page  are  both  at  the  beginning  of  the 
stack,  hence  the  induction  statement  is  still  true. 
Suppose  the  induction  statement  is  true  at  time  t,  i.e 
^  found  within  stack  distance 

2k  within  S^. 
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Suppose  the  next  reference  is  to  .  There  are 
three  cases: 

From  the  induction  statement,  is  found  within  stack 
distance  2k  in  as  illustrated  in  Fugure  4.9. 

Consider  the  page  movements  in  the  two  stacks  as  a  result 
of  handling  the  reference  to  P^  : 

W3. 

(1)  P^  and  p£  are  both  moved  to  the  top  of  their 

yfcL  yf 

stack,  the  induction  statement  still  holds  for 
these  pages. 

(2)  Each  page  in  A  increases  its  stack  distance  by 

1,  but  its  corresponding  page  is  in  A',  each  page 
of  which  can  at  most  increase  its  stack  distance 
by  1.  Thus  the  induction  statement  holds  for  all 
pages  in  A. 

(3)  None  of  the  pages  in  B  are  moved.  None  of  the  pages 
in  B'  are  moved.  (See  Figure  4.9)  If  a  page 

in  B  has  its  corresponding  page  in  B' ,  the  induction 
statement  is  not  violated.  Suppose  a  page  in  B, 

P^^  =  S^(k)  (k>x) ,  has  its  corresponding  page, 

P^  =  S^(w)  in  A' .  Then  P^  can  at  most  increase 
its  stack  distance  by  1.  But  w<  2x  because 
P^  e  A' .  Since  2k  >  2x,  the  induction  statement 
is  not  violated. 
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Case  2;  ^  e 

-  wa  t  w  t 

Each  page  in  increases  its  stack  distance  by  1. 

Each  corresponding  page  in  can  at  most  increase 
its  stack  distance  by  2,  one  due  to  the  reference  and 
one  due  to  an  overflow  from  Hence  if 

P^^  =  S^{k)  ,  k  <m.  ,  then  P^^  =  ,  (k+1)  ,  and  P^  can 

Za  t  1  Za  'C'T’X  Z 

be  found  within  stack  distance  2 (k+1)  in  at  time 
t+1. 

Case  3;  P^  ^  M^,  P^  ^ 

-  wa  t  w  t 

As  a  result  of  the  read-through  from  M  ,  each  page 

in  is  increased  by  a  stack  distance  of  1.  That  is, 

for  k  <m. ,  P^  =  si(k)  P^  =  (k+1) . 

1  za  t  ^  za  t+1 

Each  page  in  can  at  most  increase  its  stack  distance 

by  2 ,  one  due  to  loading  the  referenced  page  and  one  due 

to  an  overflow  from  Hence,  the  page  P^  is  found 

within  stack  distance  of  2k+2  in  .  Since  max(2k+2)  = 

2ra.  <m.,  is  still  in  . 

1  —  3  z 

Q.E.D. 


COROLLARY  to  LEMMA  4.1 

m.  >2m.  V  r,  t,  (S^  (m . )  )  ^  0  =  0 

j  1  '''tj''t 

PROOF  of  COROLLARY 

For  any  P^^  in  S^,  its  corresponding  page  can  be 

Za.  U- 

found  within  stack  distance  2m^  in  S^,  and  since 
pages  in  are  unique,  the  information  in  the  last 
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page  of  is  not  found  in  S^,  i.e., 

PROOF  of  Part  (b)  r  nij  ^  2m^  ^  i^/t, 

This  follows  directly  from  LEMMA  4.1. 


(sj(m.)  ) 
t  : 


i 


n  s 


Q .  E .  D  . 
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rt  H- 


THEOREM  5 


Under  GLOBAL-LRU-SOP ,  for  any  ^ 2 ,  V  r,t,  an  over¬ 
flow  from  finds  its  corresponding  page  in  iff 

m .  >  m. 

3  1 

COROLLARY 

Under  GLOBAL-LRU-SOP ,  for  any  m^^  2,  V  r,t,  an  over¬ 
flow  from  finds  its  corresponding  page  in  iff 

V  r,t,  (nj) ^  2  ^t* 

PROOF 

This  Proof  has  two  parts  as  shown  below. 

PROOF  of  Part  (a)  ;  m^  >  m^  V  r,t,  an  overflow  from 

finds  its  corresponding  page  in 

From  LEMMA  3.4  m.>m.=^V  r,t,  (M^ ) ^  2  M^  and 

31  t  t 

(sj{m.)  )  sj:  =  0.  Suppose  the  overflow  from  M^, 

t  3  t 

P^  is  caused  by  a  reference  to  M^ .  Then  just  before 
oa 

P^  is  overflowed,  P^  exists  in  M^ .  After  the  over- 
oa  o 

flow,  P^^  finds  its  corresponding  page  still  existing 
in  M^  .  Suppose  the  overflow,  P^^  ,  is  caused  by  a 
reference  to  M^.  Then  just  before  the  overflow  from 
M^,  P^  exists  in  M^ ,and  (S^ (m . ) ) ^  H  S^  =  0  i.e. , 
the  information  in  the  last  page  of  M^  is  not  M^. 

This  means  that  the  last  page  of  M^  is  not  P^,  thus, 
the  overflow  page  P^  finds  its  corresponding  page 

03. 

still  in  M^  after  an  overflow  from  M^  occurs. 
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PROOF  of  Part  (b)  :  m ^  H  r,t,  such  that  an  overflow 

from  M*"  does  not  find  its  corresponding  page  in  . 

From  THEOREM  1,  m^  <  m^  3  r ,  t,  mJ,  then  there 

i  1  *1  ”1 

exists  P  e  M,  and  P-’  We  can  find  a  reference 

za  t  z  t 

string  such  that  at  the  time  of  the  overflow  of 
P^  from  M^,  P^  is  still  not  in  .  A  string  of 

ZSi  z 

IT 

references  to  M  will  produce  this  condition.  Then 

at  the  time  of  overflow  of  P^  ,  it  will  not  find  its 

za 

corresponding  page  in  . 

Q.E.D. 
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THEOREM  6 


Under  GLOBAL-LRU-DOP ,  for  m^^2,  V  r,t,  an  overflow 
from  finds  its  corresponding  page  in  iff  >  2m^ 


COROLLARY 

Under  GLOBAL-LRU-DOP,  for  m^^2,  V  r,t,  an  overflow 
from  finds  its  corresponding  page  in  implies 
that  V  r,t, 

PROOF 

This  proof  has  two  parts  as  shown  below. 

PROOF  of  Part  (a) ;  m^  >  2m^  V  r,t,  an  overflow  from 
finds  its  corresponding  page  in  . 

THEOREM  4  ensures  that  m^  >  2m^  V  r,t,  (M^) ^  2  M^ 

and  LEMMA  4.1  ensures  that  (S^(mj))^  =  0,  we  then 

use  the  same  argument  as  in  Part  (a)  of  THEOREM  5. 

PROOF  of  Part  (b)  ;  m  ^  <  2  m^ 3  r,t,  such  that  an  overflow 
from  M^  does  not  find  its  corresponding  page  in  M^ . 

Case  1;  <  2  m^ 

m^  <2m^=^3  r,t,  (M^)  M^  (from  the  proof  of  part  (a)  of 
THEOREM  4) .  We  then  use  the  same  argument  as  in  Part 
(b)  of  THEOREM  5. 

Case  2:  m.  =  2m. 

-  j  1 

The  reference  string  r  =  ^2a'  •'*'  ^t2m  )a' 

P^2  "  will  produce  the  following  stacks 
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(at  t=2in.+l) 

1 

si  =  (pi-.  .  .  P 


(2m^)a  '  (2in^-l)a' 


P  ) 

'  (m^+l)a^ 


^2m.'  ^2in.-l'  ^1' 

1111  1 

In  handling  the  next  reference,  to  page  pi„  . ,  , 

(2in^+l)a 

the  pages  P^^  +l)a  +1  the  same 

time,  hence  the  overflow  page  pi  from 

^  ^  (m.+l)a 

1 

does  not  find  its  corresponding  page  in  . 


Q.E.D. 
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THEOREM  7 


Let  (with  pages) ,  (with  pages)  and 
be  System  A.  Let  (with  m^'  pages),  M'^  (with 

nij  '  pages)  and  be  System  B.  Let  m^'  and 

m.'  >m..  Under  GLOBAL-LRU-SOP,  for  any  ra.  >2,  no 

j  —  j  ^  X—  ' 

MLPA  can  exist  if  m.  >m.  and  m.'  >m.'. 

D  1  D  -  1 


PROOF 

We  shall  show  that  V  r,t,  (Mj(j(Mj)  ^)  S  U(M'^)^) 

This  will  ensure  that  no  MLPA  can  exist. 

Since  m. '  >m.  and  LRU  is  used  in  M^  and  M'^,  we  can 
apply  the  LRU  stack  inclusion  property  to  obtain 

From  THEOREM  5,  we  know  that  overflows  from  M^  or 

from  M'^  always  find  their  corresponding  pages  in 

M^  and  M'^  respectively.  Since  SOP  is  used,  these 

overflows  can  be  treated  as  "no-ops".  Thus,  M^ 

and  see  the  same  reference  string  and  we  can 

apply  the  LRU  stack  inclusion  property  to  obtain 

M^  CM' 2  (since  m.'  >m.  and  LRU  is  used), 
t  —  t  3  —  D 

mJ  C  M'  ^  and  M^  C  M'  j  (M^  U  (M^)  ^)  S  (M'  J  \J  (M'  ^)  ^)  . 

Q.E.D. 
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THEOREM  8 


Let  System  A  and  System  B  be  defined  as  in  THEOREM  7. 

Let  m^'  >  m^  and  m^ '  ^  m ^ .  Under  GLOBAL-LRU-DOP ,  for 

any  m.  >  2,  no  MLPA  can  exist  if  m.  >  2m,  and  m.'>2m.'. 

1  ~  D  1  D  1 

PROOF 

We  need  the  following  preliminary  results  for  this  proof. 
LEMMA  8.1 

Let  S^  be  partitioned  into  two  disjoint  stacks,  and 
V  defined  as  follows:  W  (k)  =  S^(j,)  for  k=l,...,|w  j 

t  u  t  JC  t 

where  and  is  the  minimum  such  that 

"5  P^  e  sj;  and  P^  =  S^(j,). 

-J  za  t  za  t  k 

V^{k)  =  for  k=l,  ...,  Iv^l  where  jQ=0,  and  jj^ 


is  the  minimum  ji,  >  j,  ^  such  that  s  S^,  P^  f 

“"k  -’k-1  ’  za  t  za 

(Intuitively,  W  is  the  stack  obtained  from 

t  X  L. 

S^  by  collecting  those  pages  that  have  their  corres¬ 
ponding  pages  in  M^  such  that  the  order  of  these  pages 
in  S^  is  preserved.  is  what  is  left  of  after 

is  formed.)  Then,^^ r,t,  (a)  and 

(b)  0^  where  is  the  set  of  pages  corresponding 

to  all  the  pages  that  ever  overflowed  from  M^,  up  to 
time  t. 

PROOF  of  LEMMA  8.1 

From  THEOREM  4,  m^  >  2m^  i=^Yr,t,  (Mj)^^M^.  Thus, 

.  i  .  i 

for  each  page  in  M^,  its  corresponding  page  is  in  M^. 
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This  set  of  pages  in  is  exactly  W^,  and  W^g  by 

definition.  Since  the  conditions  for  V.  and  W.  are 

t  t 

mutually  exclusive  and  collectively  exhaustive,  the 
other  pages  in  that  are  not  in  are  by  definition 
in  V^.  Since  a  page  in  does  not  have  a  correspon¬ 
ding  page  in  its  corresponding  page  must  have  once 
been  in  because  of  Read-Through,  and  later  over¬ 
flowed  from  M^.  Thus  a  page  in  is  a  page  in  0^. 

Q.E.D. 
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LEMMA  8.2 


Any  overflow  page  from  M^  is  a  page  in 
PROOF  of  LEMMA  8.2 

From  THEOREM  4 ,  m ^  >  2m  ,t,  (Mj)"-3Mj 

From  THEOREM  6  ,  m ^  >  2m ,  t ,  an  overflow  from  M^ 

always  finds  its  corresponding  page  in  M^ 

An  overflow  from  M^  is  caused  by  a  reference  to  M  .  An 
overflow  from  M^  also  implies  that  there  is  an  overflow 
from  M^. 

Suppose  the  overflow  page  from  M^  is  P^.  Also  suppose 

P^  e  i.e.,  P^  ji  V..  We  shall  show  that  this  leads 

o  t  o  t 

to  a  contradiction. 

The  overflow  page  from  M^  is  either  P^  or  P^^(y?^o). 

u  odi  ys 

If  P^  c  P^  is  overflowed  from  M^,  THEOREM  6  is  vio- 
oa  —  o  t 

i  i 

lated  since  P„^  and  P;'  overflow  at  the  same  time  so 
oa  o 

P^^  will  not  find  its  corresponding  page  in  M^ . 

If  P^^  G  p2  is  overflowed  from  ,  THEOREM  4  is  vio- 
ya  r  o  t 

lated  since  after  the  overflow  handling,  there  exists 

a  page  p\  c  P^  in  M^  (since  P^  e  W. )  but  P^  is  no 
Ob  =  o  o  t  o 

longer  in  M^ . 

Q .  E .  D . 

LEMMA  8.3 

If  there  is  no  overflow  from  either  or  then 
Yr  ,t,  and  have  the  same  reverse  ordering. 

Two  stacks  and  are  in  the  same  reverse  ordering. 
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EQ  if  rS^(k)  =  rS^ (k)  for  l£k^min  ( | |  ,  |s^|), 
where  rS  denotes  the  stack  obtained  from  S  by  rever- 
sing  its  ordering.  By  convention,  £2  if  S^=j? 
or  =  jaf 
PROOF  of  LEMMA  8,3 

To  facilitate  the  proof,  we  introduce  the  following 
definitions: 

(1)  The  ordered  parent  stack,  (S^)^,  of  the  stack 

is  the  stack  of  parent  pages  corresponding  to, 
and  in  the  same  ordering  as,  the  pages  in  the 
reduced  stack,  S^,  of  S^,  Formally,  (S^) ^ 
and  (S^)^  2 

(2)  Define  a  new  binary  operator,  concatenation  (||), 

1  2 

between  two  stacks,  S  and  S  ,  to  produce  a  new 
stack,  S,  as  follows: 

S  =  S^||S^,  where  S(k)=  fs^(k)  for  k=l,  2,...,|S^1 

S^(k)  for  k=js^j+l,..., 
^ds^l  +  Is^D 

(3)  Define  a  new  binary  operator,  ordered  difference 
(o) ,  between  a  stack  and  a  set  T,  to  produce 
a  new  stack,  S,  as  follows: 

S  =  o  T,  where  S(k)=S^(jj^)  for  k=l,2,..., 

(|S^|  -  \S^f)T\  ), 

such  that  jQ=0f  is  the  minimum  such 
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that  =  0.  Intuitively,  S  is  obtained 

from  by  taking  away  those  elements  of  which 
are  also  in  T, 

Figure  4.10  illustrates  the  LRU  ordering  of  all  Level  i 
pages  ever  referenced  up  to  time  t.  Since  there  is  no 
overflow  from  either  of  M'^,  the  length  of  this 
LRU  stack  is  less  than  or  equal  to  min (mj ,  m ^ ' ) 

By  the  definition  of  ^t  ~  ^^t^ ^ 

But  (S'^)^  =  (S^)3  11  (  (X^)^  o  (S^)^), 

hence  =  (Y^)  ^  o  (  (S^)^  IK  (x^)  ^  o  (sj)^)) 

=  (Y^)^  o  ((sJ)^g(X^)3) 

Similarly,  by  the  definition  of  V^,  =  (Z^)^o  (S^) ^ 

But  (Z^)^  =  (X^)^  II  (  (Y^)^  o 

Hence  =  ( (X^)  ^o  (S^)  ^ )  H  ( { (Y^)  ^o  (X^)^)  o  (S^)^) 

=  ((X^)3o(sj)^)  II  ((Y^)^o((S^)^g  (X^)^)) 

=  ((x^)^  o  (sj)^) 1 |v^ 

Thus,  the  two  stacks  are  in  the  same  reverse  ordering. 

Q.E.D. 
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Most 

recently 

referenced 


t -  - \r 


Least  recently 
referenced 


Figure  4.10 


LEMMA  8.4 


(a)  M'^2M^,  (b)  and  are  either  in  the 

same  reverse  ordering  or  the  last  element  of  is 
not  an  element  of 
PROOF  of  LEMMA  8.4 

(a)  and  (b)  are  true  for  any  time  before  there  is 
any  overflow  from  either  M^  or  M'^.  (a)  is  true 

because  any  page  ever  referenced  is  in  Level  j ,  so 
a  page  found  in  M^  is  also  found  in  M'^.  (b)  is 

true  because  of  the  result  from  LEMMA  8.3. 

Assume  that  (a)  and  (b)  is  true  for  t.  Consider 
the  next  reference  at  t+1.  Suppose  this  reference 
does  not  produce  any  overflow  from  either  M^  or  M'^, 
then  (a)  still  holds  because  M'^  5  ^'t  —  ^t 

(See  THEOREM  7) .  (b)  still  holds  because  overflows 

from  M^  and  M'^  are  taken  from  the  end  of  stacks 
and  respectively,  and  since  there  is  no  over¬ 
flow  from  Level  j,  (b)'s  validity  is  not  disturbed. 
Suppose  this  reference  does  produce  overflow (s) 
from  Level  j . 

Case  1  ;  overflow  from  M'^,  no  overflow  from  M^  ; 

This  cannot  happen  since  overflow  from  M'^  implies 
reference  to  M  which  in  turn  implies  overflow 
from  M^  also. 
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Case  2  :  overflow  from  ,  no  overflow  from  : 

•  Suppose  the  last  element  in  is  not  an  element 

of  V^.  Then  starting  from  the  end  of  if  we 

eliminate  those  elements  not  in  V^,  the  two 
stacks  will  be  in  the  same  reverse  ordering. 

This  follows  from  LEMMA  8.3  and  is  illustrated 
in  Figure  4.11.  Thus  we  see  that  overflow  from 
M^ ,  i.e.,  overflowing  the  last  page  of  V^,  will 
not  violate  (a)  since  this  page  is  still  in  V^. 
(b)  is  still  preserved  since  the  last  page  in 

is  still  not  in  V^. 

•  Suppose  and  are  in  the  same  reverse  order¬ 
ing.  Then  overflowing  the  last  page  of  does 
not  violate  (a)  and  results  in,.,the  last  page  of 

V'  not  in  V. . 
t  t 

Case  3  :  overflow  from  M^  and  overflow  from  M'^  : 

•  Suppose  the  last  element  in  is  not  in  V^. 
Referring  to  Figure  4.11  in  Case  2,  we  see  the 
result  of  overflowing  the  last  element  of  and 
the  last  element  of  does  not  violate  (a)  and 
still  preserves  the  condition  that  the  last 
element  of  is  not  in 

•  Suppose  and  are  in  the  same  reverse  order¬ 

ing.  Then  overflowing  the  last  elements  of 
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and  leaves  and  still  in  the  same 
reverse  ordering.  (a)  is  not  violated  since 
the  same  page  is  overflowed  form  and  . 

Q.E.D. 


PROOF  of  THEOREM  8 

for  the  same  reasons  as  those  used  in 


THEOREM  7. 

From  LEMMA  8.4  M'^  pM^. 

Hence,  (M^  ^  (M^)^)  (M'^  \)  {M'j)^) 

Q.E.D. 
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4 . 5  SUMMARY 


We  have  developed  a  formal  model  of  a  data  storage  hier¬ 
archy  system  specifically  designed  for  very  large  databases. 
This  data  storage  hierarchy  makes  use  of  different  page 
sizes  across  different  storage  levels  and  maintains  multiple 
copies  of  the  same  information  in  different  storage  levels 
of  the  hierarchy. 

Four  classes  of  algorithms  obtained  from  natural  exten¬ 
sions  to  the  LRU  algorithm  are  formally  defined  and  studied 
in  detail.  Key  properties  of  data  storage  hierarchy  systems 
that  make  use  of  these  algorithms  are  identified  and  for¬ 
mally  proved. 

It  is  found  that  for  the  LOCAL-LRU  algorithms,  no  choice 
of  sizes  for  the  storage  levels  can  guarantee  that  a  lower 
storage  level  always  contains  all  the  information  in  the 
higher  storage  levels.  For  the  GLOBAL-LRU  algorithms,  by 
choosing  appropriate  sizes  for  the  storage  levels,  we  can 
(1)  ensure  the  above  multi-level  inclusion  property  to  hold 
at  all  times  for  any  reference  string,  (2)  guarantee  that  no 
extra  page  references  to  lower  storage  levels  are  generated 
as  a  result  of  handling  overflows,  and  (3)  guarantee  that  no 
multi-level  paging  anomaly  can  exist. 
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Chapter  V 

DESIGN  OF  THE  DSH-11  DATA  STORAGE  HIERARCHY  SYSTEM 

5.1  INTRODUCTION 

In  chapter  3,  DSH-1,  a  general  structure  of  the  INFOPLEX 
data  storage  hierarchy  system  is  introduced.  DSH-1  incorpo¬ 
rates  novel  features  to  enhance  its  reliability  and  perfor¬ 
mance.  Many  alternative  architectures  of  data  storage  hie¬ 
rarchies  can  be  derived  from  DSH-1.  These  structures  can  be 
used  to  perform  detail  studies  of  various  design  issues  con¬ 
cerning  data  storage  hierarchies. 

This  chapter  describes  a  simple  structure  of  the  INFOPLEX 
data  storage  hierarchy  derived  from  DSH-1.  This  structure 
is  called  DSH-11  and  is  used  to  develop  detail  protocols  for 
supporting  the  read  and  write  operations.  Multi-level 
inclusion  properties  of  DSH-11  are  then  discussed. 

5-2  STRUCTURE  OF  DSH-11 

The  general  structure  of  DSH-11  is  illustrated  in  Figure 
5.1.  There  are  h  storage  levels  in  DSH-11,  denoted  by  L(l), 
L(2),  ...  ,  L{h).  L(l)  is  the  highest  performance  storage 
level.  L(l)  consists  of  k  memory  ports.  Each  memory  port 
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Figure  5.1  Structure  of  DSH-11 
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is  connected  to  a  processor.  All  k  memory  ports  share  the 
same  local  bus.  A  storage  level  controller  (SLC)  couples 
the  local  bus  to  the  global  bus.  The  global  bus  connects 
all  the  storage  levels. 

All  other  storage  levels  have  the  same  basic  structure. 

In  each  storage  level,  there  is  an  SLC  which  couples  the 
local  bus  to  the  global  bus.  There  is  a  memory  request  pro¬ 
cessor  (MRP)  that  handles  requests  to  the  storage  level. 
There  are  a  number  of  storage  device  modules  (SDM's)  that 
store  the  data  within  the  storage  level.  The  SLC,  MRP,  and 
SDM's  share  the  same  local  bus. 

The  number  of  SDM's  in  different  storage  levels  may  be 
different.  The  type  of  storage  device  in  the  SDM's  in  dif¬ 
ferent  storage  levels  are  different.  For  high  efficiency, 
the  block  sizes  of  different  storage  levels  are  different. 
L(l)  has  a  block  size  of  q(l),  L{2)  has  a  block  size  of 
q (2) =n (1 ) *q(l) ,  and  so  on,  where  n(l),  n(2),  ...,  n(h-l)  are 
non-zero  integers. 

5.2.1  The  DSH-11  Interface 

From  the  point  of  view  of  a  processor  connected  to  a 
DSH-11  memory  port,  DSH-11  appears  as  a  very  large  virtual 
memory  with  2**V  bytes.  The  entire  virtual  address  space  is 
byte  addressable.  The  instructions  for  a  processor  are 
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stored  in  a  local  memory  outside  of  DSH-11.  This  local 
memory  has  an  address  space  of  2**G  bytes.  Hence,  the 
effective  data  address  space  is  (2**V-2**G)  bytes. 

All  operations  on  data  within  DSH-11  are  performed  in 
L(l).  Thus,  if  a  referenced  data  item  is  not  in  L(l),  it 
has  to  be  brought  into  L{1)  from  a  lower  storage  level. 

This  induces  a  delay  on  the  instruction  comparable  to  a  page 
fault  in  virtual  memory  systems.  Each  processor  has  multi¬ 
ple  register  sets  to  support  efficient  multiprogramming. 

The  key  registers  for  interfacing  with  DSH-11  are  the  memory 
operation  register  (MOR) ,  the  memory  address  register  (MAR) , 
the  memory  buffer  register  (MBR) ,  the  operation  status 
register  (OSR) ,  and  the  process  identification  register 
(PIR) .  The  MAR  is  V  bits  wide  and  the  MBR  is  n  bytes  wide, 
where  n  is  less  than  q(l),  the  block  size  of  L(l). 

A  read  operation  requests  n  bytes  of  data  at  location 
pointed  to  by  MAR  to  be  brought  into  the  MBR.  A  write  oper¬ 
ation  requests  the  n  bytes  of  data  in  the  MBR  be  written  to 
the  location  pointed  to  by  the  MAR.  We  shall  assume  that 
the  n  bytes  of  data  in  a  memory  reference  do  not  cross  a 
L(l)  block  boundary  (If  a  data  item  crosses  block  boundar¬ 
ies,  multiple  requests  will  be  used  so  that  each  request 
only  reference  data  within  block  boundaries) . 
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A  read  or  write  operation  may  proceed  at  the  speed  of  the 
L{1)  devices  when  the  referenced  data  is  found  in  L(l). 
Otherwise,  the  operation  is  interrupted  and  the  processor 
switches  to  another  process  while  DSH-11  moves  the  refer¬ 
enced  data  into  L(l)  from  a  lower  storage  level.  When  the 
data  is  copied  into  L(l),  the  processor  is  again  interrupted 
to  complete  the  operation. 

5.2.2  The  Highest  Performance  Storage  Level  -  L(^) 

As  illustrated  in  Figure  5.1,  L{1)  consists  of  k  memory 
ports.  Each  memory  port  consists  of  a  data  cache  controller 
(DCC)  and  a  data  cache  duplex  (DCD) .  A  DCC  communicates 
with  the  processor  connecting  to  the  memory  port.  A  DCC 
also  maintains  a  directory  of  all  data  in  the  DCD.  All  k 
memory  ports  share  a  local  bus.  The  SLC  serves  as  a  gateway 
for  communication  between  L(l)  and  other  storage  levels. 

5.2.3  A  Typical  Storage  Level  -  L{^) 

A  typical  storage  level,  L(l),  consists  of  a  number  of 
SDM's,  an  MRP,  and  an  SLC.  An  SLC  serves  as  the  gateway  for 
communication  among  storage  levels.  The  MRP  services 
requests  to  L(i).  An  SDM  performs  the  actual  reading  and 
writing  of  data.  An  SDM  consists  of  a  device  controller  and 
the  actual  storage  device. 
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To  gain  high  throughput,  communications  over  the  global 
bus  is  in  standard  size  packets.  The  packet  size  is  such 
that  it  is  sufficient  for  sending  one  L(l)  data  block.  Com¬ 
munications  over  a  local  bus  at  L(i)  is  also  in  standard 
size  packets.  The  size  of  a  packet  depends  on  the  storage 
level  and  is  chosen  so  that  a  packet  is  sufficient  to  send  a 
data  block  of  size  q(i). 

In  the  following  sections,  the  read  and  write  operations 
are  discussed  in  detail. 

^•3  ALGORITHMS  FOR  SUPPORTING  THE  READ  OPERATION 
5.3.1  The  Read-Through  Operation 

A  read  request  is  issued  by  a  processor  to  its  data  cache 
controller.  The  data  cache  controller  checks  its  directory 
to  see  if  the  requested  data  is  in  the  data  cache.  If  the 
data  is  found  in  the  data  cache,  it  is  retrieved  and 
returned  to  the  processor.  If  the  data  is  not  in  the  data 
cache,  a  read-through  request  is  queued  to  be  sent  to  the 
next  lower  storage  level,  L(2),  via  the  storage  level  con¬ 
troller  . 

At  a  storage  level,  a  read-through  request  is  handled  by 
the  memory  request  processor.  The  memory  request  processor 
checks  its  directory  to  determine  if  the  requested  data  is 
in  one  of  the  storage  device  modules  at  that  level.  If  the 
data  is  not  in  the  storage  level,  the  read-through  request 
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is  queued  to  be  sent  to  the  next  lower  storage  level  via  the 
storage  level  controller. 

If  the  data  is  found  in  a  storage  level,  L(i),  a  block 
containing  the  data  is  retrieved  by  the  appropriate  storage 
device  module  and  passed  to  the  storage  level  controller. 

The  storage  level  controller  broadcasts  the  block  to  all 
upper  storage  levels  in  several  standard  size  packets.  Each 
upper  storage  level  has  a  buffer  to  receive  these  packets. 

A  storage  level  only  collect  those  packets  that  assemble 
into  a  sub-block  of  the  appropriate  size  that  contains  the 
requested  data.  This  sub-block  is  then  stored  in  a  storage 
device  module. 

At  L(l),  the  sub-block  containing  the  requested  data  is 
stored,  and  the  requested  data  is  sent  to  the  processor  with 
the  proper  identification. 

Figure  5.2  illustrates  the  read-through  operation. 

Assume  that  DSH-11  has  only  three  storage  levels,  L(l)  with 
block  size  b,  L(2)  with  block  size  2b,  and  L{3)  with  block 
size  4b.  Suppose  a  reference  to  a  data  item  'x'  is  found  in 
a  block  in  L(3).  Then  the  sub-block  of  size  2b  containing 
'x'  is  broadcasted  to  L(2)  and  L(l)  simultaneously.  L(2) 
will  accept  and  store  the  entire  sub-block  of  size  2b.  L(l) 
will  only  accept  and  store  a  block  of  size  b  that  contains 
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'x'.  The  two  sub-blocks,  each  of  size  2b  of  a  parent  block 
in  L(3)  are  referred  to  as  the  child  blocks  of  the  parent 
block . 

5.3.2  Overflow  Handling 

To  accomodate  a  new  data  block  coming  into  a  storage 
level  as  a  result  of  a  read-through,  an  existing  data  block 
may  have  to  be  evicted.  The  block  chosen  for  eviction  is 
that  which  is  the  least  recently  referenced  block  such  that 
none  of  its  child  blocks  is  in  the  immediate  upper  storage 
level.  To  support  this  scheme,  each  block  is  associated 
with  an  Upper  Storage  Copy  Code  (USC-code) .  If  any  of  the 
child  blocks  of  a  data  block  is  in  the  immediate  upper  sto¬ 
rage  level,  its  USC-code  is  set.  Each  time  the  last  child 
block  in  L(i)  of  a  parent  block  in  L(i+1)  is  evicted  from 
L(i),  an  overflow  request  is  sent  to  L(i+1)  to  reset  the 
USC-code  of  the  parent  block. 

Blocks  in  L(l)  and  L(2)  are  handled  slightly  differently 
due  to  the  use  of  data  caches  in  L(l).  Since  multiple 
copies  of  the  same  data  block  may  be  in  different  data 
caches,  evicting  a  block  does  not  necessarily  guarantee  no 
other  copy  exists  in  L(l).  The  following  strategy  is 
adopted  to  overcome  this  difficulty.  During  a  read-through , 
the  USC-code  of  the  block  in  L(2)  is  incremented  by  1.  Each 
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time  a  block  in  L(l)  is  evicted,  an  overflow  request  is  sent 
to  L{2)  to  decrement  the  USC-code  of  the* corresponding  par¬ 
ent  block.  This  strategy  does  not  requires  communications 
among  different  data  caches. 

5.3.3  Pathological  Cases  of  Read-Through 

The  parallel  and  asynchronous  operations  of  DSH-11  and 
the  use  of  buffers  at  each  storage  level  complicates  algor¬ 
ithms  for  handling  the  read  operation.  Pathological  cases 
that  affect  the  algorithms  are  discussed  below. 

5. 3. 3.1  Racing  Requests 

Two  different  requests  Rl  and  R2  may  reference  the  same 
block  of  data.  Furthermore,  these  two  requests  may  be 
closed  to  each  other  such  that  both  may  be  reading-through 
the  same  block  of  data  at  some  storage  level.  Since  a  data 
block  is  transmitted  in  several  packets  asynchronously,  each 
packet  must  be  appropriately  identified  to  avoid  confusion 
when  assembling  the  data  sub-blocks  at  higher  storage  lev¬ 
els  . 

A  similar  situation  arises  when  Rl  and  R2  are  closed 
together  such  that  R2  begins  to  read-through  the  same  data 
block  that  has  just  been  read-through  by  Rl.  Thus  a  data 
block  arriving  at  a  storage  level  may  find  that  a  copy  of  it 
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already  exists  at  that  storage  level.  In  this  case,  the 
incoming  data  block  is  ignored.  At  L(l),  this  data  block  is 
ignored  and  the  one  in  the  data  cache  is  read  and  returned 
to  the  processor,  since  this  is  the  most  recent  copy  of  the 
data  block. 

5. 3. 3. 2  Erronous  Overflow 

When  a  data  block  is  evicted  from  L(i)  to  make  room  for 
an  incoming  data  block,  an  overflow  request  containing  the 
virtual  address  of  the  evicted  data  block  may  be  generated. 
The  purpose  of  the  overflow  request  is  to  inform  L(i+1)  that 
there  is  no  longer  any  data  block  in  L(i)  with  the  same 
family  address  as  the  virtual  address  in  the  overflow 
request.  Hence,  an  overflow  request  is  generated  only  when 
the  last  member  of  a  family  in  L(i)  is  evicted. 

The  overflow  request  has  to  be  forwarded  to  the  MRP  at 
L(i+1).  At  any  point  on  the  way  to  the  MRP,  a  data  block  in 
the  same  family  as  the  evicted  block  may  be  read-through 
into  L(i).  This  poses  the  danger  that  when  the  MRP  receives 
the  overflow  request  indicating  that  no  data  block  in  the 
same  family  as  the  evicted  block  exists  in  L(i),  there  is 
actually  one  such  block  being  placed  in  L(i). 

The  following  strategy  is  incorporated  in  the  algorithms 
that  support  the  read-through  operation  to  avoid  such  a 
potential  hazard,, 


161 


1.  At  the  time  the  overflow  request  is  to  be  created 
in  L{i) ,  a  check  is  made  to  see  if  there  is  any 
data  block  in  the  same  family  as  the  evicted  block 
that  is  currently  in  any  buffer  of  L(i)  waiting  to 
be  placed  in  L{i).  If  so,  the  overflow  request  is 
not  created. 

2.  At  the  time  a  new  data  block  arrives  in  L(i),  any 
overflow  request  with  the  same  family  address  as 
the  incoming  data  block  waiting  to  be  sent  to 
L(i+1)  is  purged. 

3.  When  an  overflow  request  arrives  at  L(i+1)  from 
L(i),  a  check  is  made  to  see  if  there  is  any  data 
block  waiting  to  be  sent  to  L{i)  that  has  the  same 
family  address  as  the  overflow  request.  If  so, 

overflow  request  is  purged. 

4.  At  the  time  a  request  is  generated  to  send  a  data 
block  to  L(i),  any  overflow  request  from  L(i)  that 
is  still  waiting  in  L(i+1)  that  has  the  same 
family  address  as  the  data  block  to  be  sent  to 
L{i)  is  purged. 


5. 3. 3. 3  Overflow  to  a  Partially-assembled  Block 

Suppose  that  as  a  result  of  a  read-through  from  L(i+2), 
B(i),  the  only  child  block  of  B(i+1),  is  in  L(i)  and  B(i+1) 
is  partly  assembled  in  the  buffer  in  L{i+1).  It  is  possible 
that  B(i+1)  is  still  partly  assembled  in  L(i+1)  when  B(i)  is 
evicted  from  L(i).  The  overflow  request  will  find  that  the 
corresponding  parent  data  block  is  still  being  assembled  in 
L(i+1).  A  solution  to  this  difficulty  is  to  check,  at  the 
time  of  arrival  of  the  overflow  request,  if  there  is  any 
incoming  data  block  which  is  the  target  parent  block  of  the 
evicted  block  as  indicated  in  the  overflow  request.  If  so, 
the  overflow  request  is  held  till  this  parent  block  has  been 
placed  in  a  storage  device. 
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5.3.4  Transactions  to  Handle  the  Read  Operation 

A  read  request  is  issued  by  a  processor  to  its  data  cache 
controller.  If  the  data  is  not  found  in  the  data  cache,  it 
has  to  be  brought  up  via  a  read-through.  The  read-through 
operation  is  realized  via  a  number  of  transactions.  The 
flow  of  transactions  to  support  the  read-through  operation 
is  illustrated  in  Figure  5.3. 

^  read-through  transaction  is  created  by  a  data  cache 
controller  and  propagated  to  lower  storage  levels.  At  each 
storage  level,  the  read-through  transaction  is  handled  by  a 
memory  request  processor  which  checks  its  directory  to  see 
if  the  requested  data  is  in  the  current  storage  level.  If 
the  data  is  not  in  the  current  storage  level,  the  read- 
through  transaction  is  sent  to  the  next  lower  storage  level. 

Suppose  that  the  data  requested  is  found  at  L{i).  The 
read-through  transaction  is  terminated  and  a  retr ieve  tran¬ 
saction  is  created  to  read  the  data  from  a  storage  device. 

A  read-response-out  transaction  that  contains  the  read  data 
is  sent  to  the  storage  level  controller.  The  storage  level 
controller  generates  a  number  of  read-response-packet  tran¬ 
sactions  which  are  broadcasted  to  all  higher  storage  levels. 
Each  of  these  transactions  contains  a  sub-block  of  the 
requested  data. 
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•  Figure  5.3  Transactions  to  Support 
Read  Through 
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At  each  higher  storage  level,  an  appropriate  number  of 
read-response-packet  transactions  are  assembled  into  a 
read-response- in  transaction  which  contains  a  data  block  of 
the  appropriate  size.  The  memory  request  processor  obtains 
free  space  for  the  new  data  block  in  the  read-response-in 
transaction  either  by  using  existing  free  space  or  by  evict¬ 
ing  an  existing  block.  Eviction  of  an  existing  data  block 
may  result  in  an  overflow  transaction  being  sent  to  the  next 
lower  storage  level.  At  the  memory  request  processor,  the 
read- response- in  transaction  is  serviced  and  a  store  tran¬ 
saction  is  created.  A  storage  device  module  handles  the 
store  transaction  by  writing  the  data  to  a  storage  device. 

The  following  subsections  describe  the  algorithms  for 
handling  each  of  the  above  transactions. 

5. 3. 4.1  The  read-through  Transaction 

The  read-through  transaction  is  created  by  a  data  cache 
controller  and  propagated  down  the  storage  levels  via  the 
storage  level  controllers.  It  has  the  following  format: 

(  read-through ,  virtual-address,  process-id), 
where  virtual-address  is  the  virtual  address  of  the  refer¬ 
enced  data  item,  and  process-id  consists  of  a  CPU  identifier 
and  a  process  number.  It  is  the  identifier  of  the  process 
that  generated  the  read  operation.  The  transaction  is  han- 
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died  by  a  memory  request  processor  using  the  following 
algor ithm. 


1.  Search  directory  for  read-through. virtual-address . 

2.  If  not  found,  forward  the  transaction  to  the  sto¬ 
rage  level  controller,  which  will  send  it  to  the 
next  lower  storage  level. 

3.  If  found,  suppose  it  is  in  the  i-th  directory 
entry,  and  suppose  directory(i) .TRANSIT-code  is 
not  set,  do: 

i)  Set  directory (i) .use-code  to  indicate  a 
child  block  exists  in  the  higher  storage 
level  for  this  block.  If  this  is  level 
L(2),  increment  directory (i) .USC-code 
instead  of  setting  it. 

ii)  Set  directory ( i) . HOLD-code  to  forbid  any 
overflow  to  this  block  while  the  data  is 
being  retrieved. 

iii)  Create  a  retrieve  transaction  : (  retrieve, 
virtual-address ,directory ( i) . real-address ,p- 
rocess-id) . 

iv)  Send  the  retrieve  transaction  to  the  appro¬ 
priate  storage  device  module. 

4.  If  found,  suppose  it  is  in  the  i-th  directory 
entry,  and  suppose  directory{i) .TRANSIT-code  is 
set,  then  hold  the  request  and  retry  later.  When 
the  TRANSIT-code  is  set,  it  indicates  that  the 
corresponding  data  block  is  in  transit,  hence  any 
reference  to  it  is  not  allowed. 

5.  End. 


5. 3. 4. 2  The  retrieve  Transaction 

The  retrieve  transaction  is  created  by  a  memory  request 
processor  and  handled  by  a  storage  device  module  as  follows. 
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1.  Read  the  data  block  using  retrieve. real-address. 

2.  Create  a  read-response-out  transaction  : {  read- 
response-out  ,  virtual-address,  process-id,  data), 
where  data  is  the  block  containing  the  referenced 
data  item. 

3.  Send  the  re ad -response-out  transaction  to  the  sto¬ 
rage  level  controller. 

4.  End. 


5. 3. 4. 3  The  read-response-out  Transaction 

The  read-response-out  transaction  is  created  by  a  storage 
device  module  and  handled  by  a  storage  level  controller 
using  the  following  algorithm. 


1.  Purge  any  incoming  overflow  transaction  that  has 
the  same  family  address  as  read-response- 

out  .vir  tual-addr  ess. 

2.  Send  {  update-directory,  vir tual-addr ess , HOLD- 
code=0)  to  memory  request  processor  to  reset  the 
HOLD-code,  so  that  overflow  to  this  block  is  now 
allowed. 

3.  Broadcast  the  transaction  (  reserve -space ,  virtu¬ 
al-address,  process-id)  to  all  higher  storage  lev¬ 
els  to  reserve  buffer  space  for  assembling 

read- response-out .data . 

4.  Wait  till  all  higer  levels  have  acknowleged  the 
space  reservation  transaction. 

5.  Generate  the  appropriate  number  of  (  read-res¬ 
ponse-packet  ,  virtual-address,  process-id,  data, 
data-virtual-address)  transactions.  Data  is  a 
standard  size  sub-block  and  data-virtual-address 
is  the  virtual  address  of  this  sub-block. 

6.  Broadcast  each  read-response-packet  transaction  to 
all  higher  storage  levels. 

7.  End. 
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5. 3. 4. 4  The  read-response-packet  Transaction 


This  transaction  is  created  by  a  storage  level  controller 
and  broadcasted  to  all  higher  storage  level  controllers 
where  they  are  assembled  into  read-response- in  transactions 
to  be  handled  by  the  memory  request  processors.  Note  that  a 
storage  level  only  accepts  those  packets  relevant  for  assem¬ 
bling  into  a  data  block  of  the  appropriate  size,  all  other 
associated  packets  are  ignored.  The  following  algorithm  is 
used  by  a  storage  level  controller  in  assembling  the 
read- response- in  transactions. 

1.  If  this  is  the  first  packet  of  the  assembly,  do: 

i)  Purge  any  outgoing  overflow  transaction  that 
has  the  same  family  address  as  the  block 
being  assembled. 

ii)  Add  the  packet  to  the  assembly. 

2.  If  this  is  an  intermediary  packet  of  the  assembly, 

simply  add  it  to  the  assembly. 

3.  If  this  is  the  last  packet  of  the  assembly,  do: 

i)  Replace  the  assembly  by  a  (  read-response- 
in,  virtual-address,  process-id,  data)  tran¬ 
saction.  Data  is  the  block  just  assembled. 

ii)  Send  the  above  transaction  to  the  memory 
request  processor. 

4.  End. 
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5. 3. 4. 5  The  read-response-in  Transaction 

This  transaction  is  created  by  a  storage  level  controller 
and  sent  to  a  data  cache  controller  (for  L{1))  or  to  a 
memory  request  processor  (for  L{2),  L(3),  ...)• 


The  following  algorithm  is  used  by  a  data  cache  control¬ 
ler  in  handling  this  transaction. 


1.  Purge  any  outgoing  overflow  transaction  that  has 
the  same  family  address  as  the  block  in  the  read- 
response-in  transaction. 

2.  Search  directory  for  read-response-in. virtual- 
address  . 

3.  If  found,  suppose  it  is  the  i-th  directory  entry, 
do; 

i)  Read  data  from  the  data  cache  using  direc- 
tory(i) .real-address. 

ii)  Send  data  to  the  processor. 

iii)  Increment  directory (i) .USC-code  by  1. 

4.  If  not  found,  do: 

i)  Select  a  block  to  be  evicted  (assuming  that 
data  cache  is  full) .  This  is  the  least 
recently  referenced  block  such  that  it  is 
not  engaged  in  a  stored-behind  process. 
Suppose  this  block  corresponds  to  direc¬ 
tory  ( i)  . 

ii)  Obtain  d irectory ( i) . virtual-address ,  direc¬ 
tory  (  i)  .  USC-code  ,  and  directory(i) .real- 
address  . 

iii)  Write  read-response-in. data  into  location 
directory ( i) .real-address  in  the  data  cache. 

iv)  Return  read-response-in. data  to  the  proces¬ 
sor  . 
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V) 

Create  (  overflow,  directory(i) .virtual- 
address,  USC-code=directory ( i ) .USC-code) 
transaction,  send  it  to  the  storage  level 
controller . 

vi) 

Update  directory(i) .virtual-address 
read-response- in. virtual-address. 

with 

vii ) 

Set  d irectory ( i) . USC-code  to  1. 

5.  End. 

At  a  memory  request  processor,  the  read-response- in  tran¬ 
saction  is  handled  as  follows. 


1.  Purge  any  outgoing  overflow  transaction  with  the 
same  family  address  as  the  data  block  in  the 
read-response- in  transaction. 

2.  Search  for  r ead-r esponse-in . v ir tual-addr ess  in  the 
directory. 

3.  If  not  found,  do: 

i)  Select  a  block  to  be  evicted  (assuming  that 
the  storage  level  is  full) .  This  is  the 
least  recently  referenced  block  such  that  it 
is  not  engaged  in  a  store-behind  process,  it 
is  not  held  (i.e.,  HOLD-code  =  0),  and  it  is 
not  in  transit  (i.e.,  TRANSIT-code  =  0). 
Suppose  this  block  corresponds  to  direc¬ 
tory  ( i)  . 

ii)  Obtain  d irectory ( i ). virtual-address  and 
directory(i) .real-address. 

iii)  If  the  evicted  block  is  the  last  of  its 
family  in  the  storage  level  and  that  there 
is  no  incoming  block  with  the  same  family 
address  then  create  a  (  overflow,  direc¬ 
tory  (  i  ).  virtual-address  ,  USC-code=l)  tran¬ 
saction.  Send  the  transaction  to  the  sto¬ 
rage  level  controller  to  be  sent  to  the  next 
lower  storage  level. 

iv)  Set  directory (i) .TRANSIT-code  to  1  to  indi¬ 
cate  the  corresponding  block  is  in  transit. 
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v)  Update  directory (i) .virtual-address  with 
read-response- in .virtual-address . 

vi)  Set  directory ( i) .use-code  to  1. 

vii)  Create  a  (  store^  directory { i) . real-address , 
data)  transaction  and  send  it  to  the  appro¬ 
priate  storage  device  module. 

4.  End. 

5. 3. 4. 6  The  store  Transaction 

This  transaction  is  handled  by  a  SDM.  Store. data  is 
placed  in  store. location,  and  a  transaction  (  update-direc¬ 
tory,  virtual-address,  TRANSIT-code  =0)  is  sent  to  the  MRP 
to  reset  the  TRANSIT-code  so  that  references  to  this  block 
is  now  allowed. 

5. 3. 4. 7  The  overflow  Transaction 

This  transaction  is  created  by  a  data  cache  controller  or 
a  memory  request  processor  and  routed  to  a  memory  request 
processor  in  the  lower  storage  level  via  the  storage  level 
controllers.  At  each  stop  on  the  way  to  a  memory  request 
processor,  a  check  is  made  to  see  if  any  incoming  data  block 
has  the  same  family  address  as  the  overflow  transaction.  If 
so,  the  following  algorithm  is  executed. 

1.  If  the  direction  of  flow  of  the  overflow  and 
read-response-in  are  opposite,  the  overflow  is 
purged. 

2.  If  the  direction  of  flow  of  the  overflow  and  the 
read-response-out  are  opposite,  the  overflow  is 
purged . 
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3.  If  the  two  transactions  are  in  the  same  direction 
of  flow,  the  overflow  is  held  to  be  processed 
after  the  read-response-in  is  handled. 

At  a  memory  request  processor,  if  the  HOLD-code  is  set 
for  the  parent  block  of  the  overflowed  block,  the  overflow 
transaction  is  purged  (HOLD-code  is  set  indicates  that  the 
block  is  being  retrieved  by  an  SDM  to  be  read-through  to  all 
upper  storage  levels).  Otherwise,  the  USC-code  of  the  par¬ 
ent  block  is  decremented  by  over  flow. USC-code . 

5.4  ALGORITHMS  TO  SUPPORT  THE  WRITE  OPERATION 

Algorithms  to  support  the  write  operation  are  simplified 
by  the  multi-level  inclusion  properties  of  DSH-11.  The  mul¬ 
ti-level  inclusion  properties  of  DSH-11  guarantee  that  all 
the  data  items  in  L(i)  is  contained  in  L(i+1).  Thus,  when 
writing  a  child  block  in  L(i)  to  its  parent  block  in  L(i+1), 
the  parent  block  is  guaranteed  to  exist  in  L(i+1).  The  mul¬ 
ti-level  inclusion  properties  of  DSH-11  will  be  discussed  in 
a  later  section. 

5.4.1  The  Store-Behind  Operation 

After  a  block  is  placed  in  a  data  cache  as  a  result  of  a 
read-through  operation,  its  parent  block  exists  in  L(2),  and 
its  grand-parent  block  exists  in  L(3),  and  so  on.  Due  to 
the  multi-level  inclusion  properties  of  DSH-11,  this  situa¬ 
tion  will  persist  as  long  as  the  block  is  in  the  data  cache. 
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After  a  data  block  in  a  data  cache  is  updated,  it  is  sent 
down  to  the  next  lower  storage  level  to  replace  the  corres¬ 
ponding  child  block  in  its  parent  block.  This  updated  par¬ 
ent  block  is  sent  down  to  the  next  lower  storage  level  to 
update  its  parent  block,  and  so  on.  This  process  is  refered 
to  as  the  store-behind  operation  and  takes  place  at  slack 
periods  of  system  operation. 

DSH-11  uses  a  two-level  store-behind  strategy.  This 
strategy  ensures  that  an  updated  block  will  not  be  consid¬ 
ered  for  eviction  from  a  storage  level  until  its  parent  and 
grand-parent  blocks  are  updated.  This  scheme  will  ensure 
that  at  least  two  copies  of  the  updated  data  exists  in 
DSH-11  at  any  time.  To  support  this  scheme,  a  Store-Behind 
Code  (SB-code)  is  associated  with  each  data  block  in  a  sto¬ 
rage  level.  The  SB-code  indicates  the  number  of  acknow¬ 
ledgements  from  lower  storage  levels  that  the  block  must 
receive  before  it  can  be  considered  for  eviction. 

In  a  write  operation,  the  data  item  is  written  into  the 
data  cache  duplex,  and  the  processor  is  notified  of  the  com¬ 
pletion  of  the  write  operation,  we  shall  assume  that  the 
data  item  to  be  written  is  already  in  L{1)  (This  can  be 
realized  by  reading  the  data  item  into  L(l)  before  the  write 
operation) .  A  store-behind  operation  is  next  generated  by 
the  data  cache  controller  and  sent  to  the  next  lower  storage 
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Figure  5.4(b)  Illustration  of  Store  Behind  (b) 
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llustration  of  Store  Behind  (c) 


level.  The  block  in  L(l)  that  has  just  been  updated  is 
marked  with  a  count  of  2.  This  is  illustrated  in  Figure 
5.4  (a)  . 

When  a  store-behind  operation  is  received  in  L(2),  the 
addressed  data  is  written,  and  marked  with  a  count  of  2.  An 
acknowledgement  is  sent  to  the  next  upper  storage  level, 
L(l),  and  a  store-behind  operation  is  sent  to  the  next  lower 
storage  level,  L(3).  When  an  acknowledgement  is  received  at 
L(l),  the  counter  for  the  addressed  data  item  is  decremented 
by  1,  which  becomes  1.  This  is  illustrated  in  Figure 
5.4  (b)  . 

The  store-behind  is  handled  in  L(3)  by  updating  the 
appropriate  data  block.  An  acknowledgement  is  sent  to  L(2). 
At  L(2),  the  corresponding  block  counter  is  decremented  by 
1,  which  becomes  1.  The  acknowledgement  is  forwarded  to 
L(l).  At  L(l),  the  corresponding  block  counter  is  decre¬ 
mented  by  1  which  now  becomes  0,  hence  the  block  is  elligi- 
ble  for  replacement.  This  is  illustrated  in  Figure  5.4(c). 

Thus  we  see  that  the  two-level  store-behind  strategy 
maintains  at  least  two  copies  of  the  written  data  at  all 
times.  Furthermore,  lower  storage  levels  are  updated  at 
slack  periods  of  system  operation,  thus  enhancing  perfor¬ 
mance.  Detail  algorithms  for  supporting  this  scheme  will  be 
discussed  in  a  later  section. 
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5.4.2  Lost  Updates 

Several  different  updates  to  the  same  block  will  result 
in  several  different  store-behind  requests  be  sent  to  the 
next  lower  storage  level.  It  is  possible  that  these  store- 
behind  requests  arrive  at  the  next  storage  level  out  of 
sequence,  resulting  in  lost  updates. 

To  resolve  this  potential  hazard,  there  is  a  time-stamp 
associated  with  each  block  indicating  the  last  time  the 
block  was  updated.  There  is  also  a  time-stamp  associated 
with  each  child  block  of  the  parent  block  indicating  the 
last  time  the  child  block  was  updated  by  a  store-behind 
operation.  A  store-behind  request  will  contain  the  block  to 
be  updated  and  its  time-stamp.  This  time-stamp  will  be  com¬ 
pared  with  that  of  the  corresponding  child  block  in  the  tar¬ 
get  parent  block.  Only  when  the  store-behind  data  is  more 
recent  will  the  update  to  the  target  parent  block  be  per¬ 
formed. 

5.4.3  Transactions  to  Support  the  Write  Operation 

Figure  5.5  illustrates  the  transactions  to  support  the 
write  operation.  We  shall  assume  that  the  target  block  of  a 
write  operation  already  exists  in  a  data  cache.  This  can  be 
ensured  by  first  reading  the  target  block  before  issuing  the 
write  request  to  the  data  cache.  After  the  data  is  written 
into  a  target  data  block  in  a  data  cache,  a  store-behind 
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transaction  containing  the  updated  block  is  sent  to  the  next 
lower  storage  level.  The  store-behind  transaction  is  ser¬ 
viced  by  the  memory  request  processor.  The  memory  request 
processor  generates  an  update  transaction  and  sends  it  to 
the  appropriate  storage  device  module.  The  memory  request 
processor  also  sends  an  ack-store-behind  transaction  to  the 
higher  storage  level.  The  storage  device  module  handles  the 
update  transaction  by  replacing  the  corresponding  child 
block  in  the  target  parent  block  with  the  data  in  the 
store-behind  transaction.  Another  store-behind  transaction 
containing  the  updated  parent  block  is  created  and  sent  to 
the  storage  level  controller  to  be  forwarded  to  the  next 
lower  storage  level. 

A  store-behind  transaction  is  sent  to  the  next  lower  sto¬ 
rage  level  in  several  standard  size  packets,  each  corres¬ 
ponds  to  a  store-behind-packet  transaction.  At  a  storage 
level  controller,  these  packets  are  assembled  into  the  ori¬ 
ginal  store-behind  transaction.  The  algorithms  for  sending 
and  assembling  packets  are  very  similar  to  those  used  for 
the  read-through  operation  and  will  not  be  repeated  here. 

The  following  describes  the  algorithms  for  supporting  the 
above  transactions  to  realize  the  write  operation. 
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5.4. 3.1  The  store-behind  Transaction 

A  store-behind  transaction  has  the  following  format: 

^  store-behind ,  virtual-address , process-id , data , time-stamp) 
This  transaction  is  handled  by  a  memory  request  processor 
using  the  following  algorithm. 


1.  Search  directory  for  store-behind . virtual-address . 

2.  If  not  found,  hold  the  transaction  and  retry  after 
a  time  out,  because  the  target  parent  block  is 
still  being  assembled  in  the  buffer. 

3.  If  found,  compare  store-behind . time-stamp  with  the 
time-stamp  of  the  corresponding  child  block  of  the 
target  parent  block. 

4.  If  store-behind .data  is  more  current  than  the 
child  block,  do: 

i)  Send  (  update ,  virtual-address,  data,  real- 
address,  tirae-stamp-of-parent)  to  the  appro¬ 
priate  storage  device  module. 

ii)  Update  the  time-stamp  of  the  child  block 
with  store-behind , time-stamp. 

iii)  Send  (  ack -store-behind ,  virtual-address, 
process-id,  ACK-code  =  2)  to  the  immediate 
higher  storage  level.  ACK-code  indicates 
the  number  of  levels  this  transaction  is  to 
be  routed  upwards. 

5.  If  store-behind .data  is  not  more  current  than  data 
in  storage  level,  send  two  {  ack-store-behind , 
virtual-address,  process-id,  ACK-code  =  2)  to  the 
immediate  higher  storage  level. 

6.  End. 
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5. 4. 3. 2  The  update  Transaction 

The  update  transaction  is  handled  by  a  storage  device 
module  using  the  following  algorithm. 


1.  Replace  the  appropriate  child  block  in  the  target 
parent  block  by  update. data. 

2.  The  updated  target  parent  block  is  retrieved. 

3.  Send  {  update-directory ^  virtual-address,  SB-code 
=2)  to  the  memory  request  processor  to  increment 
SB-code  of  the  target  parent  block  by  2. 

4.  {  store-behind,  virtual-address,  process- 
id  ,  target-parent-block,  time-stamp  = 

update . time-stamp-of-parent)  is  sent  to  the  sto¬ 
rage  level  controller  to  be  sent  to  the  next  lower 
storge  level. 

5.  End. 


5. 4. 3. 3  The  ack-store-behind  Transaction 

This  transaction  is  handled  by  a  memroy  request  proces¬ 
sor.  The  algorithm  used  is  as  follows. 

1.  The  SB-code  of  the  corresponding  block  is  decre¬ 
mented  by  1. 

2.  The  ack-store-behind .ACK-code  is  decremented  by  1. 

3.  If  ack-store-behind .ACK-code  is  greater  than  0  the 
forward  the  ack-store-behind  to  the  immediate 
upper  storage  level. 

4.  End. 
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5.5  MULTI-LEVEL  INCLUSION  PROPERTIES 

As  a  result  of  the  read-through  operation,  the  block 
read-through  into  L(l)  leaves  its  'shadow'  in  every  lower 
storage  level  that  participated  in  the  read-through  opera¬ 
tion.  Is  it  true  then,  that  a  storage  level,  L(i),  always 
contains  every  data  block  in  L(i-l)?  When  this  is  true, 
multi-level  inclusion  (MLI)  is  said  to  hold. 

It  has  been  formally  proved  in  Chapter  4  that  certain 
algorithms  incorporating  the  read-through  strategy  can  guar¬ 
antee  MLI  provided  that  the  relative  sizes  of  the  storage 
levels  be  appropriately  chosen.  Furthermore,  it  is  found 
that  certain  other  algorithms  can  never  guarantee  MLI.  This 
section  explores  the  MLI  properties  of  DSH-11.  In  the  fol¬ 
lowing  sections,  the  importance  of  MLI  is  briefly  reviewed, 
a  model  of  DSH-11  is  developed,  and  the  MLI  property  of 
DSH-11  is  analyzed  informally. 

5.5.1  Importance  of  MLI 

The  MLI  properties  have  important  implications  for  the 
performance  and  availability  of  DSH-11.  First ,  since  the 
block  size  of  L(i)  is  larger  than  that  of  L(i-l),  L(i)  can 
be  viewed  as  an  extension  of  the  spatial-locality  (Madnick, 
1973)  of  L(i-l).  Second ,  except  for  the  lowest  storage 
level,  each  data  item  has  at  least  two  copies  in  different 
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storage  levels.  Hence,  even  the  failure  of  an  entire  sto¬ 
rage  level  will  not  result  in  data  loss.  Third ,  algorithms 
to  support  the  write  operation  is  simplified  if  MLI  holds 
because  a  store-behind  operation  always  finds  the  target 
parent  data  block  exists  in  a  storage  level. 

5.5.2  A  Model  of  DSH-11 

Figure  5.6  illustrates  a  model  of  DSH-11.  DSH-11  has  h 
storage  levels,  L(l),  L(2),  ...  ,  L{h).  L(l)  consists  of  k 
data  caches.  Each  data  cache  consists  of  a  buffer  B(l,i), 
and  a  storage,  M(l,i).  All  the  buffers  of  the  data  caches 
are  collectively  denoted  as  B(l),  and  all  the  storage  of  the 
data  caches  are  collectively  denoted  as  M(l).  The  size  of 
B(l,i)  is  b(l,i)  number  of  blocks  of  size  q(l).  The  size  of 
M(l,i)  is  m(l,i)  number  of  blocks  of  size  q(l).  Hence  L{1) 
has  b{l)  =  b(l,l)  +  b(l,2)  +  ...  +  b{l,k)  blocks  of  buffer 
space  and  m(l)  =  m(l,l)  +  m(l,2)  +  ...  +  m(l,k)  blocks  of 
storage  space. 

A  buffer  is  for  holding  data  blocks  coming  into  or  going 
out  of  the  storage  level.  A  data  block  may  be  partially 
assembled  in  a  buffer.  Only  data  blocks  in  the  storage 
space  are  accounted  for  by  the  directory.  Note  that  a  buf¬ 
fer  is  not  used  for  holding  transactions  that  do  not  contain 
any  data,  e.g.  an  ack-store-behind  transaction  does  not 
occupy  any  buffer  space. 
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A  typical  storage  level,  L(i),  consists  of  a  buffer  B(i), 
and  a  storage,  M(i).  The  size  of  B(i)  is  b(i)  number  of 
blocks  each  of  size  q(i).  The  size  of  M(i)  is  m(i)  number 
of  blocks  each  of  size  q(i) .  The  block  size  of  L{i)  is 
q(i),  where  q(i)  =  n ( i-1 ) *q ( i-1) ,  for  i  =  2,  3,  ...  ,  h. 

The  n(i)'s  are  integers. 

5.5.3  Properties  of  DSH-11 

Based  on  the  model  in  the  previous  section,  the  MLI  con¬ 
dition  can  be  stated  as  follows:  a  data  block,  whole  or 
partially  assembled,  that  is  found  in  L(i)  is  also  found  in 
L(i+1).  This  section  shows  that  for  DSH-11,  it  is  possible 
to  guarantee  the  following:  (1)  MLI  holds  at  all  times,  (2) 
it  is  always  possible  to  find  a  block  for  eviction  to  make 
room  for  an  incoming  block,  and  (3)  an  overflow  transaction 
from  L(i)  always  finds  its  target  parent  block  in  L(i+1). 

Proposition 

Let  J  =  i  +  1.  Using  the  algorithms  described  in  the  previ¬ 
ous  sections,  if  m(j)  is  greater  than  m(i)+b{i)  then 

1.  MLI  holds  for  L(i)  and  L(j),  i.e.,  any  block  found 
in  L(i)  can  be  found  in  L(j), 

2.  If  block  replacement  in  M(j)  is  required,  there  is 
always  a  block  not  in  L{i)  that  can  be  considered 
for  overflow,  and 

3.  An  overflow  transaction  from  L(i)  always  contains 
the  address  of  a  block  that  can  be  found  in  M(j). 

Proof  of  Proposition 
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There  are  three  cases: 


1.  There  are  no  overflows  from  L{i):  Since  m(j)  is 
greater  than  m(i)+b(i),  no  overflow  from  L(i) 
implies  no  overflow  from  L{j).  Thus  all  blocks 
present  in  L(i)  are  still  in  L(j),  i.e.,  (1)  is 
true. 

2.  There  are  overflows  from  L(i) ,  no  overflow  from 
L{j):  No  overflow  from  L(j)  implies  that  all 
blocks  referenced  so  far  are  still  in  L(j).  Thus 
any  block  in  L(i)  is  still  in  L(j),  i.e.,  (1)  is 
true.  Since  any  overflow  from  L(i)  will  find  the 
block  still  in  L(j),  (3)  is  true. 

3.  There  are  overflows  from  L(j);  Consider  the  first 
overflow  from  L(j).  Just  before  the  overflow,  (1) 
is  true.  Also  just  before  the  overflow,  M(j)  is 
full.  M{j)  is  full  and  m(j)  is  greater  than 
m(i)+b{i)  implies  that  there  is  at  least  one  block 
in  M{j)  that  is  not  in  L{i)  (i.e.,  their  USC-code 
=0).  Choose  from  these  blocks  the  least  recently 
referenced  block  such  that  its  SB-code  =0.  If  no 
such  block  exists,  wait,  and  retry  later.  Eventu¬ 
ally  the  store-behind  process  for  these  blocks 
will  be  terminated  and  these  blocks  will  be 
released.  Thus  a  block  will  be  available  for 
overflow  from  M(j).  Thus  (2)  is  true.  After  the 
overflow,  (1)  is  still  preserved.  (1)  and  (2) 
implies  (3) . 


If  next  reference  causes  no  overflow  from  L(j) ,  then  the 
arguement  in  Case  2  applies.  If  the  next  reference  causes 
overflow  from  L(j),  then  the  arguement  in  Case  3  applies. 


5. 6  SUMMARY 

The  DSH-11  design,  a  data  storage  hierarchy  for  the  INFO- 
PLEX  data  base  computer,  is  described.  Protocols  for  sup¬ 
porting  the  read  and  write  operations  in  DSH-11  are  des- 


187 


cribed  in  detail.  It  is  then  shown  that  DSH-11  is  able  to 
guarantee  multi-level  inclusion  at  all  times  for  any  refer 
ence  string  provided  that  the  sizes  of  the  buffers  and  sto 
rage  at  the  storage  levels  are  chosen  appropriately. 
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Chapter  VI 


SIMULATION  STUDIES  OF  THE  DSH-11  DATA  STORAGE  HIERARCHY 

SYSTEM 


6.1  INTRODUCTION 

This  chapter  discusses  the  results  of  a  series  of  simula¬ 
tion  studies  of  the  DSH-11  data  storage  hierarchy  system. 

A  key  objective  of  these  simulation  studies  is  to  assess 
the  feasibility  of  supporting  very  large  transaction  rates 
(millions  of  reads  and  writes  per  second)  with  good  response 
time  (less  than  a  millisecond)  using  the  DSH-11  storage 
hierarchy  and  the  read-through  and  store-behind  algorithms. 

A  GPSS/360  simulation  model  is  developed  for  a  DSH-11 
configuration  with  one  processor  and  three  storage  levels. 
The  results  obtained  from  this  model  are  very  interesting. 

It  is  found  that,  at  very  high  locality  levels ,  when  most  of 
the  references  are  satisfied  by  the  highest  performance  sto¬ 
rage  level,  the  store-behind  algorithm  interacts  with  the 
DSH-11  buffer  manaigement  algorithms  to  create  a  system  dead¬ 
lock.  This  has  not  been  anticipated  in  the  design  of 
DSH-11,  and  has  led  to  a  redesign  of  the  DSH-11  buffer  man¬ 
agement  scheme. 
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Another  GPSS/360  simulation  model  is  developed  for  a 
DSH-11  configuration  with  five  processors  and  four  storage 
levels.  This  model  makes  use  of  deadlock-free  buffer  man¬ 
agement  algorithms.  Results  from  this  model  reveal  further 
interesting  properties  of  the  store-behind  algorithm  and  of 
the  DSH-11  design.  It  is  found  that  at  high  locality  lev¬ 
els,  the  store-behind  requests  form  a  pipeline.  Thus  the 
rate  of  write  operations  that  can  be  serviced  is  limited  by 
the  slowest  stage  in  the  pipeline,  i.e.,  the  slowest  storage 
device.  It  is  also  found  that  a  bottleneck  may  be  developed 
at  the  lowest  level  when  the  block  size  of  that  level  is  too 
large . 

A  well-balanced  system  is  obtained  by  increasing  the 
degree  of  parallelism  in  the  lower  storage  levels  and  by 
decreasing  the  block  sizes  used  by  these  storage  levels. 

This  system  is  then  used  as  a  basis  to  compare  the  perfor¬ 
mance  of  the  DSH-11  architecture  under  different  technology 
assumptions.  It  is  found  that  using  1979  technologies,  a 
throughput  of  .7  million  operations  per  second  with  mean 
response  time  of  60  microseconds  are  obtained  for  a  mix  of 
storage  references  consisting  of  30  percent  read  requests. 
Using  1985  technologies,  the  same  storage  reference  mix  pro¬ 
duces  a  throughput  of  4  million  operations  per  second  with 
10  microseconds  mean  response  time. 
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6.2  A  SIMULATION  MODEL  OF  DSH-11  :  P1L3  MODEL 

The  P1L3  model  of  DSH-11  is  a  GPSS/360  model  of  a  DSH-11 
configuration  with  one  processor  and  three  storage  levels. 

It  represents  a  basic  structure  from  which  extensions  to 
include  more  processors  and  storage  levels  can  be  made.  The 
structure  of  P1L3  is  illustrated  in  Figure  6.1(a).  Each 
module  in  Figure  6.1(a)  actually  consists  of  four  queues  and 
a  facility  (Figure  6.1(b)).  The  facility  is  referred  to  as 
the  request  processor  (RP) .  There  are  two  input  queues,  one 
for  transactions  with  data  (the  XQ) ,  and  one  for  transac¬ 
tions  with  messages  (the  IQ) .  The  two  corresponding  output 
queues  are  named  YQ  and  OQ  respectively.  The  XQs  and  YQs 
have  limited  capacity,  since  they  are  the  data  buffers. 

There  is  no  limit  on  the  lengths  of  the  IQs  and  the  OQs . 

The  following  example  illustrates  the  naming  conventions 
used  in  the  model.  The  K2  module  actually  consists  of  the 
KRP2,  KIQ2,  K0Q2,  KXQ2  and  KYQ2.  The  current  length  of  KXQ2 
is  denoted  as  KXL2  and  the  maximum  allowable  length  of  KXQ2 
is  denoted  as  KXM2. 

6.2.1  ^  Illustration  of  the  DSH-11  Algorithms 

A  listing  of  the  P1L3  model  is  presented  in  Appendix  A. 

To  illustrate  the  model  logic,  the  following  is  a  brief  des¬ 
cription  of  the  path  followed  by  a  read-through  transaction. 
A  read  request  (TXN)  is  queued  in  KIQ3  (the  input  message 
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Figure  6.1(a)  The  P1L3  Configuration 


XQ  YQ 


Figure  6.1(b)  A  Module  in  P1L3 
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DEGREE  OF  MULTI  PROGRAMING  OF  A  CPU  =  20 

SIZES  OF  DATA  QUEUES  (XQ  AND  YQ)  =  10 

DIRECTORY  SEARCH  TIME  =  200  NANOSEC. 

read/write  time  of  a  L(1)  storage  device  =  100  NANOSEC. 

read/write  time  of  a  L(2)  device  =  1000  nanosec. 

read/write  time  of  a  L(3)  device  =  10000  nanosec. 

BUS  SPEED  =  10  MHZ 
BUS  WIDTH  =  8  BYTES 

SIZE  OF  A  transaction  WITHOUT  DATA  =  8  BYTES 
BLOCK  SIZE  AT  L(l)  =  8  BYTES 
BLOCK  SIZE  AT  LO  =  128  BYTES 
BLOCK  SIZE  AT  L(3)  =  1024  BYTES 
%  READ  REQUESTS  =  70% 

%  WRITE  REQUESTS  =  30% 

CONDITIONAL  PROB.  OF  FINDING  DATA  IN  A  LEVEL 
GIVEN  THAT  THE  DATA  IS  NOT  IN  ANY  UPPER  LEVEL  =  P 


Figure  6.2  Input  Parameters  to  P1L3 


queue  of  the  storage  level  controller  at  level  3) .  When 
KRP3  is  free,  TXN  is  serviced  and  put  into  K0Q3.  When  LBUS3 
is  available,  TXN  is  sent  to  RIQ3  (the  input  message  queue 
of  the  memory  request  processor  at  level  3)  where  it  waits 
for  RRP3,  the  request  processor.  RRP3  then  searches  its 
directory  to  obtain  the  real  address  for  TXN.  TXN  is  put 
into  R0Q3  to  be  sent  to  a  storage  device,  say,  D31.  When 
LBUS3  is  free,  TXN  is  sent  to  DIQ31  (the  input  message  queue 
for  device  D31) .  TXN  waits  in  DIQ31  for  DRP31  to  be  free 
and  also  for  a  slot  in  DYQ31  (the  output  data  queue  for  D31) 
to  hold  the  retrieved  data.  When  both  conditions  are  met, 
DRP31  retrieves  the  data  and  puts  it  in  DYQ31  where  it  waits 
for  the  LBUS3  to  be  free  and  for  there  to  be  a  slot  in  KXQ3 
(the  input  data  queue  of  the  storage  level  controlJer  at 
level  3)  to  hold  the  data.  VJhen  both  conditions  are  met, 
the  data  is  sent  to  KXQ3.  Then  the  data  is  put  in  KYQ3 
waiting  for  the  GBUS  and  for  all  the  upper  storage  levels  to 
be  free  to  receive  the  broadcast... 

6.2.2  The  P1L3  Model  Parameters 

The  model  is  highly  parametized.  Parameters  for  the  P1L3 
model  are  chosen  to  reflect  current  (1979)  processor  and 
storage  technology.  A  key  parameter  that  characterizes  the 
references  made  to  DSH-11  is  the  locality  level .  The  local¬ 
ity  level  (P)  is  the  condition  probability  that  a  reference 
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is  satisfied  at  a  given  storage  level  given  that  the 
reference  is  not  satisfied  in  all  upper  storage  levels. 

Figure  6.2  summarizes  all  the  model  parameters. 

6.3  SIMULATION  RESULTS  OF  THE  P1L3  MODEL 

Three  different  locality  levels  are  used  for  the  P1L3 
model.  The  simulated  time  is  one  milisecond  (one  million 
time  units  in  the  model) .  Some  unusual  phenomena  are  uncov¬ 
ered  during  the  analysis  of  these  preliminary  results.  This 
leads  to  more  extensive  simulation  studies  to  obtain  more 
data  points.  A  plausible  theory  is  then  proposed  to  explain 
these  phenomena.  Detail  traces  of  the  model  is  used  to  ver¬ 
ify  the  theory.  The  findings  are  discussed  in  the  following 
subsections . 

6.3.1  Preliminary  Studies  Using  the  P1L3  Model 

A  series  of  three  simulation  studies  are  carried  out  with 
three  locality  levels  ;  high  (P=.85),  medium  {P=.5),  and  low 
(P=.2).  Throughputs,  mean  response  times  and  utilizations 
of  the  facilities  are  summarized  in  Figure  6.3. 

Throughput  in  millions  transactions  per  second  are  plot¬ 
ted  against  the  locality  levels  in  Figure  6.4.  From  Figure 
6.4,  it  seems  that  a  throughput  of  .6  million  transactions 
per  second  is  the  maximum  that  one  could  obtain  with  this 
configuration. 
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Figure  6.3  Preliminary  Results  of  the  P1L3  Model 


Locality  Level 


Throughput  per  millisecond 


GBUS  Utilization 
LBUSl  Utilization 


Data  Cache  Utilization 

LBUS2  Utilization 

D21  Utilization 

LBUS3  Utilization 
D31  Utilization 


Mean  Response  Time  (Nsec) 
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Figure  6.5  Mean  Response  Time  Vs.  Locality  Level 
(  P1L3  Preliminary  Results) 
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Figure  6.6  Utilization  Vs.  Locality  Level 
(P1L3  Preliminary  Results) 


Mean  response  time  per  transaction  are  plotted  against 
locality  levels  in  Figure  6.5.  This  shows  that  a  mean  res¬ 
ponse  time  of  5  micro  seconds  is  obtainable  at  high  locality 
levels.  Furthermore,  as  the  locality  level  increases,  there 
will  be  more  references  being  satisfied  in  the  high  perfor¬ 
mance  storage  levels,  thus  the  mean  response  time  will 
decrease . 

Utilizations  of  the  various  facilities  are  plotted 
against  locality  levels  in  Figure  6.6.  It  can  be  seen  from 
these  plots  that  at  low  locality  levels,  the  slowest  storage 
level  becomes  a  system  bottleneck.  At  higher  locality  lev¬ 
els,  bus  utilizations  drop  because  most  references  are 
satisfied  by  the  data  cache,  DRPll,  making  the  use  of  the 
buses  unnecessary  except  for  store-behind  operations. 

At  high  locality  levels,  one  would  also  expect  the  utili¬ 
zation  of  the  data  cache,  DRPll,  to  be  very  high.  However, 
this  is  not  supported  by  the  data.  In  fact,  even  though  the 
throughput  at  the  P=.85  locality  level  is  larger  than  that  ■ 
at  the  P=.50  locality  level,  the  DRPll  utilization  actually 
drops . 

Examine  the  data  more  closely,  another  puzzle  is  discov¬ 
ered.  If  one  multiply  throughput  by  the  mean  response  time 
divided  by  the  maximum  degree  of  multiprogramming,  one 
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should  obtain  a  number  closed  to  the  simulated  time 
(1,000,000).  For  the  P=.20  case,  this  number  comes  out  to 
be  915657.  For  the  P=.50  case,  this  number  comes  out  to  be 
858277.  But  for  the  P=.85  case,  this  number  is  only  180027. 
It  is  suspected  that  either  the  data  is  wrong  or  there  are 
some  unusual  blocking  phenomena  in  the  system  in  the  P=.85 
case . 

2  More  Extensive  Studies  Using  the  P1L3  Model 

Since  it  is  not  difficult  to  obtain  more  data  points  by 
varying  the  locality  levels,  a  second  series  of  simulations 
is  carried  out.  The  results  of  these  simulations  are  pre¬ 
sented  in  Figure  6.7. 

Throughputs  are  plotted  against  locality  levels  in  Figure 
6.8.  This  shows  that  as  the  locality  level  increases, 
throughput  also  increases.  A  throughput  of  closed  to  one 
million  transactions  per  second  is  obtainable  at  about  P=.80 
locality  level.  However,  after  the  P=.80  point,  throughput 
drops  sharply  as  the  locality  level  increases.  This 
requires  some  explaination. 

In  Figure  6.9,  mean  response  time  is  plotted  against 
locality  levels.  This  shows  that  as  locality  level 
increases,  mean  response  time  decreases.  This  plot  does  not 
seem  to  provide  insight  as  to  why  throughput  decrease  shar¬ 
ply  after  the  P=.80  locality  level. 
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Figure  6.7  P1L3  Model  -  Extensive  Results 
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Figure  6.8  Throughput  Vs.  Locality  Level 
{P1L3  Extensive  Results) 
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Figure  6.9  Mean  Response  Time  Vs.  Locality  Level 
(P1L3  Extensive  Results) 


6.3.3  A  Plausible  Theory  of  Operation 

One  theory  to  explain  the  sharp  drop  in  throughput  at 
very  high  locality  levels  is  that  at  such  high  locality  lev¬ 
els,  the  rate  of  write  operations  being  generated  is  very 
high.  Since  a  write  will  not  proceed  until  DRPll  is  free 
(to  write  the  data),  and  DRPll 's  YQ  has  a  buffer  slot  (for 
holding  the  store-behind  operation),  the  write  operation  may 
hold  up  other  transactions  in  their  use  of  DRPll.  Since  the 
utilization  of  DRPll  is  very  low,  the  blocking  must  be  due 
to  the  YQ  being  full  often.  Many  store-behind  transactions 
closed  togther  will  tend  to  make  the  YQ  full  often.  These 
blocking  transactions  will  tend  to  hold  up  other  transac¬ 
tions  hence  resulting  in  low  system  throughput. 

If  the  YQ  is  full  often,  it  must  be  because  transactions 
in  it  cannot  move  on  to  the  next  facility.  This  will  happen 
if  the  bus  LBUSl  is  busy  or  the  XQ  buffer  of  Kl  is  full,  or 
both.  From  the  data,  we  see  that  all  the  bus  utilizations 
are  very  low,  hence  the  blocking  must  be  due  to  the  fact 
that  the  XQ  buffer  of  Kl  is  full  often.  Proceeding  in  this 
manner,  one  could  argue  that  at  high  locality  levels,  the 
rate  of  store-behind  operations  is  very  high,  which  results 
in  store-behind  transactions  being  backed  up  from  a  storage 
device.  This  backing  up  of  store-behind  operations  causes 
long  queueing  delays  for  other  transactions  as  well,  result- 
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ing  in  low  system  throughput.  This  blocking  situation  also 
prevents  DRPll  to  be  kept  busy  as  evident  from  its  low  uti¬ 
lization. 

We  can  now  explain  why  the  utilization  of  DRPll  at  the 
P=.85  locality  level  is  lower  than  that  at  the  P=.50  local¬ 
ity  level.  At  P=.85,  due  to  the  store-behind  transactions 
being  backed  up,  very  few  acknowledgements  to  the  store-be¬ 
hind  transactions  ever  return  to  DRPll.  In  the  P=.50  case, 
most  acknowledgements  to  store-behind  transactions  return  to 
DRPll.  Thus,  even  though  the  number  of  reads  and  writes 
handled  by  DRPll  in  the  P=.50  case  is  lower  than  that  han¬ 
dled  by  the  DRPll  in  the  P=.85  case,  there  are  many  more 
acknowledgements  serviced  by  DRPll  in  the  P=.50  case,  hence 
the  corresponding  utilization  is  higher. 

There  are  no  backing  up  of  store-behind  transactions  in 
the  low  locality  level  cases  because  the  rate  at  which  they 
are  generated  is  low.  Since  the  store-behind  transactions 
are  separated  from  one  another  there  is  enough  time  for  a 
device  to  service  a  previous  store-behind  transaction  before 
another  one  comes  along. 


206 


6.3.4  Verification  of  Theory  with  Data 

The  above  theory  seems  to  explain  the  phenomena  well  and 
agrees  well  with  the  observed  data.  To  verify  the  theory, 
detail  model  traces  are  examined  to  determine  the  status  of 
the  system  at  the  time  of  simulation  termination. 

It  is  found  that  for  low  locality  levels,  there  is  indeed 
no  backing  up  of  the  store-behind  transactions.  There  is  a 
backlog  of  requests  to  be  processed  by  the  lowest  storage 
level  devices  due  to  their  large  service  times.  For  high 
locality  levels,  starting  from  P=.85,  store-behind  transac¬ 
tions  begin  to  be  backed  up,  from  storage  level  2.  However, 
the  back  up  is  due  to  a  system  deadlock  developed  at  storage 
level  2,  and  not  due  to  the  slower  speeds  of  the  devices,  as 
hypothesized  above. 

The  deadlock  at  storage  level  2  is  illustrated  in  Figure 
6.10.  All  the  XQs  and  YQs  are  full.  A  store-behind  tran¬ 
saction  in  DYQ21  is  waiting  for  LBUS2  and  a  KXQ2  buffer 
slot.  LBUS2  is  free  but  KXQ2  buffer  is  full.  KXQ2  will  not 
be  cleared  because  KYQ2  is  full.  KYQ2  cannot  be  cleared 
because  both  buffer  of  R2  are  full.  These  buffers  cannot  be 
cleared  because  DXQ21  and  DYQ21  are  full.  DYQ21  cannot  be 
cleared  because  it  is  waiting  for  KXQ2  to  be  cleared.  Thus 
a  deadlock  is  developed.  This  deadlock  causes  the  XQs  and 
YQs  in  the  upper  storage  levels  to  be  gradually  filled  as 
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Store-behind 


Figure  6.10  Deadlock  In  P1L3  Model 
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more  store-behind  transactions  are  generated.  When  YQ  at 
DRPll  is  full,  the  system  will  be  held  up  when  the  next 
write  transaction  arrives. 

It  is  interesting  to  note  that  this  deadlock  only  occurs 
at  very  high  locality  levels.  This  is  beacuse  at  high 
locality  levels,  the  rate  of  store-behind  transactions  gen¬ 
erated  is  very  high.  Comparing  the  P=.95  case  and  the  P=.50 
case,  even  though  the  same  number  of  store-behind  transac¬ 
tions  are  generated  to  lower  storage  levels  in  both  cases, 
the  rate  at  which  they  are  generated  in  the  P=.95  case  is  30 
times  that  of  the  P=.50  case.  Store-behind  transactions 
sparsely  separated  from  one  another  give  chance  for  the  dev¬ 
ice  to  service  them,  therefore  avoiding  a  deadlock.  This 
deadlock  situation  is  not  too  different  from  a  traffic  jam 
at  a  Boston  rotary  during  rush  hour. 

In  retrospect,  the  causes  of  the  deadlock  are  due  to  the 
rate  of  store-behind  transactions  and  the  use  of  one  single 
buffer  for  data  coming  into  a  storage  level  as  well  as  for 
data  going  out  of  a  storage  level.  The  potential  for  dead¬ 
lock  of  using  a  common  buffer  was  not  discovered  during  the 
design  of  DSH-11  due  to  the  complex  interactions  of  the  var¬ 
ious  protocols  for  store-behind,  read-through,  and  overflow 
handling  operations. 
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6.4  DEADLOCK-FREE  BUFFER  MANAGEMENT  SCHEMES 


In  the  DSH-11  simulation  model,  there  are  five  types  of 
transactions  supporting  the  read-through  and  store-behind 
operations.  These  transactions  are  :  read-through-request 
(RR) ,  read-through-result  (RT) ,  overflow  (OV) ,  store-behind- 
request  (SB) ,  and  acknowledgement  (AK) .  Each  type  of  tran¬ 
saction  is  handled  differently.  Furthermore,  the  same  type 
of  transaction  is  handled  differently  depending  on  whether 
the  transaction  is  going  into  or  out  of  a  storage  level.  A 
potential  deadlock  exists  when  different  transactions  share 
the  same  buffer  and  their  paths  form  a  closed  loop.  We  have 
seen  an  example  of  such  deadlock  in  the  P1L3  model  where  SB 
transactions  coming  into  a  storage  level  and  SB  transactions 
going  out  of  a  storage  level  form  a  closed  loop.  Other 
potential  deadlocks  exists  in  the  P1L3  model.  This  section 
is  focused  on  developing  deadlock-free  buffer  management 
algorithms. 

Potential  deadlocks  exist  beacause  different  transaction 
types  share  the  same  buffer  and  that  the  First  Come  First 
Serve  (FCFS)  strategy  is  used  for  allocating  buffer  slots. 

A  simple  strategy  to  avoid  deadlock  is  not  to  allow  buffer 
sharing  among  different  transaction  types.  No  path  crossing 
can  occur  thus  no  loop  can  exist.  Although  this  strategy  is 
easy  to  implement,  it  does  not  make  optimal  use  of  the  buf- 
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fer  space.  Another  strategy  to  avoid  deadlock  is  to  allow 
buffer  sharing,  but  to  make  use  of  more  sophisticated  buffer 
allocation  algorithms.  One  such  algorithm  is  discussed 
below. 

6.4.1  A  Deadlock-free  Buffer  Allocation  Algorithm 

Two  types  of  buffers  are  used  at  each  storage  level,  the 
IN  buffers  and  the  OUT  buffers.  Transactions  coming  into 
the  storage  level  use  the  IN  buffers  and  transactions  going 
out  of  the  storage  level  use  the  OUT  buffers.  Transaction 
coming  into  a  storage  level  from  a  higher  storage  level  are 
the  RR,  SB,  and  OV  transactions.  Transactions  coming  into  a 
storage  level  from  a  lower  storage  level  are  the  RT  and  AK 
transactions.  Similarly,  transactions  going  out  of  a  sto¬ 
rage  level  to  the  next  lower  storage  level  are  the  RR,  SB, 
and  OV  transactions.  Transactions  going  out  of  a  storage 
level  to  a  higher  storage  level  are  the  RT  and  AK  transac¬ 
tions.  Each  component  in  a  storage  level  has  an  IN  buffer 
and  an  OUT  buffer.  This  is  illustrated  in  Figure  6.11. 

The  general  idea  of  this  buffer  allocation  scheme  is  not 
to  allow  the  buffers  to  be  completely  filled.  When  the  buf¬ 
fers  are  filled  up  to  a  certain  level,  only  those  transac¬ 
tions  that  can  be  processed  to  completion  and  resulting  in 
freeing  up  buffer  slots  are  accepted.  The  precise  algorithm 
is  as  follows. 
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Figure  6.11  A  Deadlockrf ree  Buffer  Scheme 
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1.  The  size  of  OUT  is  always  greater  than  the  size  of 
IN. 

2.  Always  maintain  at  least  one  empty  slot  in  an  IN 
buffer . 

3.  Buffer-full  (BF)  condition  is  raised  when  the  num¬ 
ber  of  transactions  in  IN  plus  the  number  of  tran¬ 
sactions  in  OUT  is  equal  to  the  size  of  OUT. 

4.  If  BP  then  do  not  accept  any  RR  or  SB  into  a  sto¬ 
rage  level.  Only  process  OV,  RT,  and  AK  transac¬ 
tions. 

We  now  provide  an  informal  argument  to  show  that  the 
scheme  described  above  is  indeed  deadlock-free.  First  we 
have  to  show  that  the  RR  and  SB  transactions  are  not  the 
only  transactions  in  the  system  when  all  the  buffer  pairs 
have  their  BF  conditions  raised.  Then  we  have  to  show  that 
processing  each  of  )the  OV,  AK  and  RT  transactions  will  free 
up  some  buffer  slots  thus  lowering  some  BF  conditions. 

Suppose  that  all  the  BF  conditions  are  raised.  Examine 
the  OUT  buffers  of  the  lowest  storage  level.  Since  the  size 
of  OUT  is  greater  than  that  of  IN,  BF  implies  that  there  is 
at  least  one  transaction  in  OUT.  This  transaction  must  be 
going  out  of  the  storage  level  to  a  higher  storage  level, 
hence  cannot  be  a  RR  or  a  SB  transaction. 
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Level  i-1 


Figure  6.12  Illustration  of  the  Deadlock-free 
Buffer  Scheme 
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Consider  a  RT  transaction  at  level  i+1  (Figure  6.12). 

(1)  All  upper  storage  level,  level  i  and  level  i-1  can 
receive  this  transaction  since  there  is  always  one  empty 
slot  in  each  IN  buffer.  The  departure  of  the  RT  transaction 
creates  an  empty  slot  in  the  OUT  buffer  of  the  sender  (level 
i+1).  (2)  Level  i  can  now  send  a  transaction  to  level  i+1 

which  creates  a  slot  in  level  i.  The  RT  transaction  can  now 
be  serviced  in  level  i.  (3)  Handling  the  RT  transaction  may 
create  an  OV  transaction  in  level  i.  Luckily  there  is  a 
buffer  slot  for  the  OV  transaction  in  level  i.  (4)  The  OV 
transaction  can  be  sent  to  level  i+1  because  there  is  always 
a  free  slot  in  the  IN  buffer  at  level  i+1.  (5)  The  OV  tran¬ 

saction  will  be  serviced  to  completion  in  level  i+1.  Hence, 
there  is  a  free  slot  in  level  i  as  result  of  these  opera¬ 
tions.  (6)  Now  a  transaction  from  level  i-1  can  be  sent  to 
level  i.  (7)  The  RT  transaction  can  be  handled  in  level  i-1 
which  may  create  an  OV  transaction.  (8)  The  OV  transaction 
can  be  sent  to  level  i.  (9)  Finally,  the  OV  transaction  is 
handled  and  terminated  in  level  i.  Thus,  there  is  a  free 
buffer  slot  created  in  level  i-1  as  a  result  of  processing 
the  RT  transaction. 

Handling  an  AK  transaction  may  generate  another  AK  to  be 
sent  to  the  immediate  upper  storage  level.  The  same  argu¬ 
ment  for  the  RT  transaction  can  be  applied  to  show  that  a 
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buffer  slot  will  be  freed  up  as  a  result  of  handling  the  AK 
transaction. 

It  is  clear  from  the  above  discussion  that  this  buffer 
management  scheme  requires  more  complex  protocols  among  sto¬ 
rage  levels  and  a  complex  priority  scheme  for  the  transac¬ 
tions.  A  key  advantage  of  this  scheme  is  that  it  makes 
efficient  use  of  buffer  space  since  different  transactions 
with  varying  buffer  space  requirements  can  share  a  common 
buffer  pool. 

6. 5  ANOTHER  SIMULATION  MODEL  OF  DSH-11  ;  Ti^  P5L4  MODEL 

A  GPSS/360  simulation  model  of  another  DSH-11  configura¬ 
tion  with  five  processors  and  four  storage  levels  is  devel¬ 
oped.  This  model  is  referred  to  as  the  P5L4  model.  This 
model  revised  the  basic  logic  used  in  the  P1L3  model  to  use 
a  deadlock-free  buffer  management  scheme  and  to  accomodate 
four  additional  processors  and  an  additional  storage  level. 
The  simple  schem^e  of  using  separate  buffers  for  different 
transactions  is  used  for  the  P5L4  model. 

The  first  series  of  studies  provides  further  insights  to 
the  operation  of  the  store-behind  algorithms.  It  also  shows 
that  level  4  storage  may  be  too  slow  and  its  local  bus  may 
not  have  engough  bandwidth  to  support  the  amount  of  data 
transfer  activities  at  that  level. 
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The  second  series  of  studies  is  aimed  at  obtaining  a 
well-balanced  system.  The  degree  of  parallelism  in  the 
lower  storage  levels  are  increased  and  the  demand  on  the 
buses  is  lowered  by  reducing  the  block  sizes.  A  well-ba¬ 
lanced  system  is  obtained  which  is  then  used  as  the  basic 
system  to  study  the  effect  of  using  projected  1985  technolo¬ 
gies  for  DSH-11.  Results  of  these  studies  and  their  analy¬ 
sis  are  presented  in  the  following  sections,  after  a  brief 
introduction  to  the  P5L4  model  and  its  parameters. 

6.5.1  The  P5L4  Model  and  its  Parameters 

The  structure  of  the  P5L4  model  is  very  similar  to  that 
of  the  P1L3  model.  However,  the  basic  component  of  the 
model  is  quite  different.  The  basic  component  of  the  P5L4 
model  is  a  facility  and  a  number  of  data  buffers,  one  for 
each  type  of  transaction  comming  into  the  storage  level  and 
going  out  of  the  storage  level.  Figure  6.13(a)  illustrates 
the  DSH-11  configuration  that  P5L4  is  modelling,  and  Figure 
6.13(b)  illustrates  the  basic  component  of  the  model.  A 
flow  chart  of  the  P5L4  model  logic  is  presented  in  Appendic 
B.  A  listing  of  the  P5L4  model  is  presented  in  Appendix  C. 

The  parameters  used  in  the  P5L4  model  are  the  same  as 
those  used  in  the  P1L3  model  with  the  following  exceptions. 
(1)  There  are  five  processors,  each  with  10  degrees  of  mul- 
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DEGREE  OF  MULTI  PROGRAMING  OF  A  CPU  =10 

SIZES  OF  DATA  BUFFERS  =  10 

DIRECTORY  SEARCH  TIME  =  200  NANOSEC. 

read/write  time  of  a  L(1)  STORAGE  DEVICE  =  100  NANOSEC. 

read/write  time  of  a  L(2)  device  =  1000  nanosec. 

read/write  time  of  a  L(3)  device  =  10000  nanosec. 

BUS  speed  =  10  MHZ 

BUS  WIDTH  =  8  BYTES 

SIZE  OF  A  transaction  WITHOUT  DATA  =  8  BYTES 
BLOCK  SIZE  AT  L(l)  =  8  BYTES 
BLOCK  SIZE  AT  1(2  =  128  BYTES 
BLOCK  SIZE  AT  L(3)  =  102^  BYTES 
%  READ  REQUESTS  =  70% 

%  WRITE  REQUESTS  =  30% 

CONDITIONAL  PRCB.  OF  FINDING  DATA  IN  A  LEVEL 
GIVEN  THAT  THE  DATA  IS  NOT  IN  ANY  UPPER  LEVEL  =  P 

Figure  6.14  Input  Parameters  to  the  P5L4  Model 
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t iprogramming  (as  opposed  to  20  in  the  P1L3  model) .  (2) 

There  is  a  new  storage  level  with  2  storage  devices  having 
access  times  10  times  higher  than  those  of  the  devices  in 
level  3.  The  parameters  used  In  the  P5L4  model  are  summar¬ 
ized  in  Figure  6.14. 

6.5.2  Preliminary  Studies  Using  the  P5L4  Model 

A  preliminary  study  using  the  P5L4  model  is  carried  out 
using  several  different  locality  levels  and  using  the  param¬ 
eters  listed  in  Figure  6.14.  The  simulated  time  is  one  mil¬ 
lisecond  (one  million  model  time  units) .  Results  from  these 
studies  are  summarized  in  Figure  6.15.  Figure  6.15(a)  is  a 
table  listing  the  throughput,  mean  response  time,  total 
transaction  wait  time,  total  transaction  execution  time,  and 
'system  utilization'.  System  utilization  is  defined  as  the 
ratio  of  the  product  of  the  total  number  of  transactions 
completed  and  the  mean  response  time  to  the  product  of  the 
simulated  time  and  the  maximum  number  of  active  requests 
pending  at  all  the  processors.  It  indicates  the  percentage 
time  that  DSH-11  is  busy. 

Figure  6.15(b)  tabulates  the  utilizations  of  the  buses 
and  the  utilizations  of  typical  storage  devices  at  each  sto¬ 
rage  level.  The  utilizations  of  all  the  memory  request  pro¬ 
cessors  and  all  the  the  storage  level  controllers  are  very 
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Figure  6.17  Mean  Response  Time  Vs.  Locality  Level 
(P5L4  Preliminary  Results) 
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low.  Figure  6.i5(b)  shows  that  the  devices  and  the  local 
bus  at  level  4  are  satuarated  for  all  locality  levels.  The 
local  bus  at  level  3  is  saturated  but  the  devices  at  level  3 
are  only  50  percent  utilized.  Saturation  of  level  4  at  low 
locality  levels  is  due  to  the  large  number  of  read-through 
requests  that  has  to  be  handled  at  that  level.  For  example, 
at  a  locality  level  of  .5,  one-fouth  of  al3.  read  requests 
will  be  serviced  by  level  4.  This  creates  a  heavy  burden  on 
the  level  4  devices  and  on  its  bus.  At  high  locality  lev¬ 
els,  however,  the  number  of  read-through  requests  directed 
to  level  4  is  rather  small.  For  example,  at  a  locality 
level  of  .8,  only  .8  percent  of  all  read  requests  are  ser¬ 
viced  by  level  i.  The  saturation  of  level  4  at  high  local¬ 
ity  levels  is  due  to  the  store-bebind  requests.  At  high 
locality  levels,  the  number  of  write  requests  are  much 
higher,  thus  there  is  a  lugh  demand  on  level  4  to  service 
the  corresponding  store-behind  requests.  It  seems  that 
level  3  storage  devices  have  the  capacity  to  handle  the 
read-through  and  store-behind  requests  at  all  locality  lev¬ 
els.  However,  the  local  bus  at  level  3  is  saturated  at  all 
locality  levels.  The  bus  saturation  at  level  3  is  possibly 
due  to  the  store-behind  requests.  We  shall  discuss  ways  to 
eliminate  these  performance  bottlenecks  in  a  later  section. 
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Throughput  data  presented  in  Figure  6.15(a)  is  plotted  as 
a  graph  in  Figure  6.16.  Figure  6.16  shows  that  throughput 
rises  sharply  starting  from  the  .5  locality  level,  then  its 
follows  a  much  slower  rate  of  increase  after  the  .7  locality 
level.  At  a  low  locality  level,  throughput  is  low  since  a 
large  proportion  of  the  read  requests  has  to  go  to  the 
slower  storage  devices.  As  the  locality  level  increases,  a 
large  proportion  of  requests  can  be  handled  by  the  higher 
storage  levels.  The  higher  storage  levels  are  not  heavily 
utilized,  thus  they  can  complete  the  requests  quickly.  The 
faster  transactions  can  be  completed,  the  faster  new  tran¬ 
sactions  can  arrive  since  the  model  is  a  closed  one.  This 
explains  the  sharp  increase  in  throughput  between  .5  and  .7 
locality  levels. 

When  the  locality  level  is  high,  the  rate  of  store-behind 
transactions  coming  into  the  model  becomes  high.  Since 
there  is  a  fixed  proportion  of  reads  and  writes  in  the 
request  stream,  the  throughput  at  high  locality  levels 
becomes  limited  by  how  fast  the  store-behind  requests  can  be 
serviced.  Thus,  at  high  locality  levels,  increasing  the 
locality  level  further  will  not  produce  a  dramatic  increase 
in  throughput. 

The  plot  of  mean  response  time  in  Figure  6.17  provides 
further  insights  to  the  store-behind  operations.  Figure 
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6.17  shows  that  thete  is  a  discontinuity  in  the  mean 
response  time  curve  between  .6  and  .7  locality  levels.  The 
discontinuity  may  be  explained  as  follows.  As  the  locality 
level  increases,  the  rate  of  store-behind  transactions  com¬ 
ing  into  the  model  also  increases.  Read  operations  become  a 
less  dominant  facto?  of  system  performance.  There  is  a 
pipeline  of  buffet  slots  for  store-behind  transactions-  A 
write  request  is  completed  as  soon  as  it  has  completed  a 
write  to  its  data  cache  and  has  placed  a  store-behind  tran¬ 
saction  in  the  store-behind  pipeline.  Tlie  store-behind 
transaction  flows  along  the  pipeline  until  it  is  serviced 
and  teimlnated  by  a  level  4  storage  device.  If  a  write 
request  cannot  i  rod  a  slot  in  the  store-behind  pipeline,  it 
has  to  wait.  At  higi’  iocaliiy  levels,  the  store-behind 
pipeline  is  full,  hence,  wri-e  operations  tend  to  incura  a 
larger  wait  time  waiting  for  pipeline  slots.  It  seems  that 
the  store-behind  pipeline  is  full  after  the  .7  locality 
level,  causing  long  wait  times  by  transactions,  hence  larger 
mean  response  times  for  all  locality  levels  higher  than  .7. 
The  store-behind  pipeline  is  rot  full  for  all  locality  lev¬ 
els  below  .7.  Thus  transactions  have  smaller  mean  response 
time  in  these  cases.  This  expains  the  difference  in  behav¬ 
ior  of  the  two  mean  response  time  curves. 
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The  data  seems  to  support  this  theory.  Outputs  from  the 
simulation  runs  shows  that  the  pipeline  is  full  for  all 
locality  levels  greater  than  and  equal  to  .7.  The  total 
transaction  time  column  in  Figure  6.15(a)  shows  that  there 
is  a  dramatic  increase  in  the  transaction  wait  time  for  all 
cases  with  locality  level  above  .7.  The  figure  also  shows 
that  the  transaction  wait  time  is  a  dominant  portion  of  the 
total  transaction  time.  Since  mean  response  time  is  the 
ratio  of  total  transaction  time  to  total  number  of  completed 
transactions,  the  more  than  doubling  of  the  wait  time  going 
from  .6  to  .7  locality  level  is  the  key  factor  in  the  sudden 
increase  in  mean  response  time.  The  sudden  increase  in  wait 
time  is  due  to  the  fact  that  the  pipeline  is  just  filled  up, 
new  transactions  begin  to  experience  prolonged  delays. 

These  preliminary  studies  have  provided  valuable  insights  to 
the  dynamics  of  the  store-behind  operation.  We  now  have 
gained  enough  understanding  of  the  model  to  tune  it  for  bet¬ 
ter  performance. 

6.5.3  Tuning  the  P5L4  Model 

Our  objective  in  this  next  series  of  studies  is  to  try  to 
obtain  a  well-balanced  system.  From  the  preliminary  stu¬ 
dies,  we  know  that  to  reduce  mean  response  time  we  have  to 
increase  the  efficiency  of  the  store-behind  pipeline.  One 
approach  to  increase  the  efficiency  of  the  pipeline  is  to 
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increase  the  parallelism  of  the  lower  storage  levels,  so 
that  the  service  times  of  the  stages  of  the  pipeline  are 
better  balanced.  The  preliminary  studies  also  reveal  that 
our  initial  choice  of  block  sizes  may  not  be  appropriate  for 
the  system. 

The  approach  that  is  taken  to  obtain  a  well-balanced  sys¬ 
tem  is  as  follows.  The  locality  level  is  fixed  at  .9.  Then 
the  degree  of  parallelism  in  level  3  is  increased  by  a  fac¬ 
tor  of  5  and  that  of  level  4  is  increased  by  a  factor  of  10. 
This  is  accomplished  by  decreasing  the  effective  service 
times  of  the  devices  at  these  levels  appropriately. 

Finally,  i;l:o  model  is  run  for  several  choices  of  block  sizes 
for  the  storage  levels.  The  simulated  time  for  these  runs 
are  much  longer  than  in  the  preliminary  studies  to  ensure 
that  steady  state  behavior  is  observed.  The  results 
obtained  are  summarized  in  Figure  6.18. 

The  first  study  uses  the  same  block  sizes  as  those  used 
in  the  preliminary  studies.  The  results  of  this  study  are 
summarized  in  column  one  which  clearly  shows  that  level  4  is 
the  bottleneck  causing  the  very  low  throughput  and  high  mean 
response  time.  Note  that  the  devices  are  not  saturated. 

This  indicates  that  the  block  sizes  are  too  large  thus  tie- 
ing  up  the  bus  at  level  4  during  data  transfer. 
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In  the  next  study,  the  size  of  data  transfer  between 
level  2  and  level  3  and  that  between  level  3  and  level  4  are 
reduced  by  one  half.  The  results  of  this  study  are  summar¬ 
ized  in  column  2.  The  bus  at  level  4  is  still  a  bottleneck. 
There  is  significant  improvement  in  the  utilizations  of 
level  4  storage  devices. 

Next,  the  size  of  data  transfer  between  level  3  and  level 
4  is  halved.  This  produces  a  fairly  well-balanced  system. 
The  results  are  summarized  in  column  3.  A  throughput  of  ,1 
million  operations  per  second  with  mean , response  time  of  60 
microseconds  is  obtained.  The  utilizations  across  storage 
levels  are  well-balanced  comparatively. 

6.5.4  Comparing  the  Performance  of  DSH-11  using  1979 
and  1985  Technologies 

The  well-balanced  system  obtained  from  the  previous  stu¬ 
dies  will  be  used  as  a  basis  for  comparing  the  performance 
of  DSH-11  under  1979  and  1985  technology  assumptions.  The 
parameters  used  in  the  1979  case  are  exactly  those  used  in 
the  well-balanced  system  of  the  previous  studies.  For  the 
1985  case,  we  will  use  a  bus  speed  that  is  5  times  faster 
than  that  used  in  the  1979  case.  In  general,  the  speeds  of 
the  storage  devices  in  the  1985  case  will  be  faster.  We 
estimate  that  the  level  1  storage  devices  will  be  twice  as 
fast  in  1985  as  in  1979.  All  other  devices  are  estimated  to 
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be  10  times  faster  in  1985  than  in  1979.  Lastly,  we  expect 
1985  to  produce  better  associative  processors  for  directory 

t 

searching  thus  the  directory  search  time  will  be  reduced  by 
one  half  in  1985.  These  estimates  will  be  incorproated  in 
the  parameters  for  the  1985  case. 

The  model  using  1979  technology  assumptions  is  run  for  4 
different  request  streams  with  different  proportions  of 
reads  and  writes.  The  model  using  1985  technology  assump¬ 
tions  is  then  run  with  the  same  4  different  request  streams. 
The  locality  level  is  fixed  at  .9  in  both  cases.  The 
results  are  summarized  in  Figure  6.19. 

The  throughputs  for  the  two  cases  are  plotted  on  the  same 

1 

graph  in  Figure  6.20.  In  general,  for  both  cases,  through¬ 
put  increases  as  the  proportion  of  read  requests  increases. 
It  can  be  inferred  from  the  results  that  the  throughput  of 
DSH-11  using  1985  technology  is  between  5  to  10  times  better 
than  using  1979  technology.  For  a  request  stream  with  70 
percent  read  requests  and  30  percent  write  requests,  DSH-11 
using  1979  technology  can  support  a  throughput  of  .7  million 
requests  per  second  with  a  mean  response  time  of  60  microse¬ 
conds.  For  the  same  mix  of  requests,  DSH-11  using  1985 
technology  can  support  a  throughput  of  4  million  requests 
per  second  with  a  mean  response  time  of  10  microseconds. 
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Figure  6.19  Comparing  Performance  of  P5L4 
Using  Different  Technologies 
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6.6  SUMMARY 


Two  simulation  models  cf  the  DSH-11  storage  hierarchy 
system  are  developed  and  used  to  understand  the  performance 
characteristics  of  DSH-11  and  its  algorithms.  The  first 
model  is  developed  for  a  DSH-11  configuration  with  one  pro¬ 
cessor  and  three  storage  levels.  Results  from  this  model 
uncovers  an  unsuspected  deadlock  potential  in  the  DSH-11 
buffer  management  scheme.  This  leads  to  the  development  of 
new  buffer  management  schemes  for  DSH-11.  A  second  model  is 
developed  for  a  DSH-11  configuration  with  five  processors 
and  four  storage  levels.  This  model  also  makes  use  of  a 
deadlock-free  buffer  management  scheme.  Results  from  this 
model  provides  much  insights  to  the  performance  implications 
of  the  read-through  and  store-behind  algorithms.  After  suf¬ 
ficient  understanding  of  the  model  is  obtained,  the  model  is 
tuned  for  better  performance.  The  resulting  system  is  then 
used  as  a  basis  for  com.paring  the  performance  implication  of 
using  different  technology  for  DSH-11. 

Results  from  these  simulation  studies  not  only  provide 
valuable  insights  to  the  important  dynamic  behavior  of 
store-behind  and  read-through  algorithms,  they  also  provide 
assurance  that  the  DSH-11  is  capable  of  supporting  the 
memory  requirements  of  the  INFOPLEX  functional  hierarchy. 
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Chapter  VII 

DISCUSSIONS  AND  CONCLUSIONS 

7.1  INTRODUCTION 

Database  management  is  a  major  component  of  computer 
usage.  Adapting  conventional  computer  architectures  to  sup¬ 
port  database  management  functions  has  several  disadvan¬ 
tages.  Two  major  disadvantages  have  been  recognized  for 
some  time.  These  are  :  (1)  processor  power  limitation  of 

the  conventional  computer,  and  (2)  the  'access  gap'  that 
exists  between  main  memory  and  secondary  storage  devices  of 
conventional  computers. 

Various  approaches  have  been  proposed  to  develop  special¬ 
ized  architectures  for  database  management.  These 
approaches  have  been  discussed  in  Chapter  1.  One  of  these 
approaches  is  the  INFOPLEX  data  base  computer  effort.  INPO- 
PLEX  eliminates  the  processor  power  limitation  by  using  mul¬ 
tiple  specialized  functional  processors  and  makes  use  of  a 
generalized  storage  hierarchy  specifically  designed  for  man¬ 
aging  very  large  databases.  A  major  obstacle  to  realize 
effective  storage  hierarchy  systems  has  been  the  lack  of 
understanding  of  these  systems  and  their  algorithms.  Previ- 
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ous  studies  of  storage  hierarchy  systems  have  been  focused 
on  systems  with  two  or  three  storage  levels,  and  usually  for 
program  storage.  This  thesis  is  focused  on  the  study  of 
generalized  storage  hierarchy  systems  for  data  storage, 
referred  to  as  data  storage  hierarchy  systems.  Theories  and 
models  of  data  storage  hierarchy  systems  are  developed. 
Formal  definitions  of  data  management  algorithms  for  data 
storage  hierarchy  systems  are  defined.  Important  properties 
of  data  storage  hierarchy  systems  have  been  analyzed  in 
detail  to  provide  valuable  insight  for  design  of  practical 
data  storage  hierarchy  systems.  Designs  for  the  INFOPLEX 
data  storage  hierarchy  are  developed  and  protocols  for  real¬ 
izing  the  read  and  write  operations  are  specified.  Finally, 
simulation  models  for  these  designs  are  developed  to  assess 
the  feasibility  of  these  designs  for  supporting  the  very 
high  transaction  rates  of  INFOPLEX  and  to  obtain  better 
understanding  of  the  read-through  and  store-behind  opera-^ 
tions  from  a  practical  point  of  view. 

7.2  SUMMARY  OF  THESIS 

Chapter  1  of  the  thesis  provides  a  framework  for  under¬ 
standing  the  rationale  behind  various  approaches  to  develop 
specialized  machines  for  data  management.  Major  contribu¬ 
tions  of  this  thesis  are  also  listed. 
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The  background  and  motivation  for  this  research  is  the 
INFOPLEX  data  base  computer  project.  Concepts  of  the  INFO- 
PLEX  data  base  computer  are  presented  in  Chapter  2.  An 
example  functional  hierarchy  and  an  example  storage  hier¬ 
archy  for  INFOPLEX  are  used  to  illustrate  some  of  these  con¬ 
cepts. 

A  preliminary  design  of  the  INFOPLEX  data  storage  hier¬ 
archy  is  proposed  in  Chapter  3.  Design  objectives  and  the 
structure  of  the  system  are  presented.  Further  design 
issues  that  need  to  be  resolved  are  also  identified. 

Formal  modelling  and  analysis  of  data  storage  hierarchy 
systems  are  presented  in  Chapter  4.  It  contains  formal 
proofs  of  the  multi-level  inclusion  (MLI),  the  multi-level 
overflow  inclusion  (MLOI),  and  multi-level  paging  anomaly 
(MLPA)  properties  of  data  storage  hierarchy  systems. 

The  preliminary  design  of  the  INFOPLEX  data  storage  hier¬ 
archy  system  presented  in  Chapter  2  is  simplified  in  Chapter 
5.  This  simplified  design  is  then  used  to  develop  protocols 
for  supporting  the  read  and  write  operations.  Specifica¬ 
tions  for  these  protocols  are  presented  in  Chapter  5. 

A  simulation  model  of  the  INFOPLEX  data  storage  hierarchy 
system  with  one  functional  processor  and  three  storage  lev¬ 
els  is  developed  in  Chapter  6.  Results  from  this  simulation 
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model  are  analyzed.  Insights  from  these  analysis  lead  to 
some  design  changes.  Another  simulation  model  of  the  INFO- 
PLEX  data  storage  hierarchy  is  then  developed.  This  model 
incorporates  five  functional  processors  and  four  storage 
levels.  Results  from  this  model  are  analyzed  and  reveal 
further  interesting  properties  of  the  design  and  of  the  data 
management  algorithms.  The  impacts  of  using  projected  1985 
technology  are  also  assessed. 

7.3  DIRECTIONS  FOR  FURTHER  RESEARCH 

This  thesis  has  provided  a  theoretic  framework  for  formal 
analysis  of  data  storage  hierarchy  systems.  Using  this 
framework,  several  important  properties  of  data  storage 
hierarchy  systems  thc^t  have  performance  and  reliability 
implications  are  studied  in  detail.  This  work  also  opens  up 
many  areas  for  further  investigation.  Do  the  properties  of 
data  storage  hierarchy  systems  proved  in  this  thesis  hold 
for  systems  using  any  stack  algorithm  (Mattson  et.  al . , 
1970)?  What  are  the  effects  of  introducing  the  two-level 
store-behind  algorithm  into  the  system?  Are  the  conditions 
for  avoiding  the  multi-level  paging  anomaly  (MLPA)  also 
necessary  conditions,  i.e.,  what  are  the  causes  of  MLPA? 
These  are  interesting  and  important  questions.  The  formal 
basis  developed  in  this  thesis  will  be  a  major  steping  stone 
toward  resolving  these  open  questions. 
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The  preliminary  design  of  the  INFOPLEX  data  storage 
hierarchy  can  be  used  to  develop  algorithms  that  improve  the 
efficiency  and  reliability  of  the  data  storage  hierarchy. 

The  automatic  data  repair  algorithms  introduced  in  Chapter  3 
are  particularly  interesting  and  promising.  A  number  of 
other  design  issues  are  discussed  but  left  as  open  issues. 
For  example,  the  multi-cache  consistency  problem  by  itself 
is  a  subject  of  great  importance  but  still  quite  lacking  of 
theoretic  basis  for  analysis. 

The  simulation  results  reveal  several  important  proper¬ 
ties  of  the  design  and  of  the  algorithms  that  are  quite 
unexpected.  The  deadlock  potential  in  the  initial  design 
can  be  corrected  quite  easily.  The  fact  that  the  store-be- 
hind  operation  can  be  a  system  bottleneck  is  not  anticipated 
before.  It  has  been  argued  in  the  past  that  store-behind 
operations  take  place  during  system  slack  periods  thus  do 
not  adversly  impact  system  performance.  A  number  of  alter¬ 
native  schemes  can  be  developed  to  improve  the  efficiency  of 
the  store-behind  operation.  May  be  we  can  separate  the  read 
only  data  from  the  read/write  data  and  keep  the  read/write 
data  higher  up  in  the  data  storage  hierarchy  system.  This 
would  reduce  the  store-behind  traffic  to  lower  storage  lev¬ 
els.  The  implications  of  this  type  of  data  management  stra¬ 
tegy  remain  uncharted. 
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LISTING  OF  THE  P1L3  MODEL 
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FILE:  GPSS1 


VS1J0D  D1 


CONYEESATIONAL  MOHITOB  SYSTEM 


//LAKI  JCtJ  L^H,  nPI!OFILE=' RETURN', 

//  PBOFILE=' HIGH'  , 

//  TinE=2 
//♦PASswcp.:? 

//C.PSS  PRCC 

//C  EXEC  PGK=DRG01 ,TiaE=2 

//STEPLIB  DD  DSN=PCTLUCK. LIBRARY. GPSS. LOAD, DISP=SHa 
//DOUTPUT  DD  SYSOtIT=PHOFILE=KETURH,  DCB=BLKSIZE  =  '931 
//DIHTEEO  DD  U NIT  =  SC BATCH, SPACE= <CYL,  ( 1 ,  1) ), DC3=BLKSI2E= 1 880 
//DSYRTAB  DD  ON IT=SCRAaCH,S?ACE=  (CYL,  (1 , 1 ) ) ,DCB=B LKSIZE=7 1 1 2 
//DHEPTGEN  DD  0 NIT=SCRATCH, SPACE=  (CYL,  ( 1, 1) ) , DC3=BLKS12E  =  800 
//DINTRCRK  DD  U NI T= SCR ATCH, SPACE=  (C YL,  (1 ,1 ) ) ,DCB*BLKSIZE=2680 
//  PEND 

//STEPI  EXEC  GPSS, PAEa=C 
//DINPOTI  DD  ♦ 

*««*««  <*!**««  **«**««**«K*«A**««*>)>**«**«»*«*«*«4<*«i|'«*****4> 

* 


TRANSACTION  PARAMETER  USAGE 

♦ 

P  1 

CPU  IDENTIFIER 

P2 

ARRIVAL  TIME 

* 

P3 

COMPLETION  TIME 

* 

P4 

TOTAL  EXECUTION  TIME 

P5 

TOTAL  ELAPSED  TIME 

t 

P6 

TOTAL  WAIT  TIME 

* 

P7 

SERVICE  TIME 

* 

P11 

DUMMY 

*****  *********************************************************** 

* 


NT  XU 

EQU 

01, X 

NUMBER  OF  TXNS  PROCESSED 

SOMX 

EQU 

02, X 

EXECUTION  TIME  OP  ALL  TXNS 

SOMQ 

EQU 

03, X 

QUEUE  TIME  OF  ALL  TXNS 

SUHH 

EQU 

04, X 

ELAPSED  TIME  OP  ALL  TXNS 

MAXMP 

EQU 

05, X 

DEGREE  OP  CPU  MU LTIPLROGB AMMING 

KREAD 

EQU 

06, X 

PARTS  IN  THOUSAND  OF  READ  TXNS 

NHRIT 

EQU 

07, X 

PARTS  IN  THOUSAND  OF  WRITE  TXNS 

P1N1 

EQU 

08, X 

PEOB  OF  FINDING  READ  DATA  IN  L(1) 

PIN2 

EQU 

09, X 

PROB  OF  FINDING  READ  DATA  IN  L(2) 

P1N3 

* 

EQU 

10, X 

PROD  OF  FINDING  READ  DATA  IN  L{3) 

P0V1 

EQU 

11, X 

PEOB  OF  OVERFLOW  FROM  L(1) 

P0V2 

EQU 

12, X 

PROD  OF  OVERFLOW  FROM  L(2) 

P0V3 

EQU 

13, X 

PROB  OP  OVERFLOW  FROM  L  (3) 

* 


***************************************************************** 

*  MAXlaUn  DATA  QUEUE  LENGTHS  ♦ 

***************************************************************** 

* 


DXM1 1 

EQU 

14, X 

DYM1  1 

EQU 

15, X 

DXH12 

EQU 

16, X 

0YH12 

EQU 

17, X 
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DXH13 

eq'u 

16, X 

0Tni3 

EQO 

19, X 

* 

DEH21 

EUU 

20, X 

DIE21 

EQO 

21, X 

DXH22 

EQU 

22, X 

D1H22 

EQU 

23, X 

* 

DXn31 

cQO 

24, X 

DYH31 

EQU 

25, X 

DXH32 

EQU 

26, X 

DtH32 

EQW 

27, X 

« 

KXH1 

EQO 

28, X 

KtHl 

EQU 

29,X 

* 

KXH2 

EQO 

30,X 

k;h2 

EQO 

31, X 

* 

KXH3 

EQO 

32, X 

KTH3 

EQO 

33, X 

* 

BXH2 

EQU 

34, X 

fiYfl2 

EQO 

35,X 

* 

BXa3 

EQO 

36, X 

BYH3 

EQO 

37, X 

* 

turn*****  ************  ***************************^**11** 

*  CUEEEilT 

LENGTHS  OP  DATA  QOEOES  • 

***************************************************************** 

* 

DXL11 

EQU 

38, X 

DYL1 1 

EQO 

39, X 

DXL12 

EQO 

40, X 

DYL12 

EQU 

41, X 

ML13 

EQO 

42,  X 

DXL13 

EQO 

43,X 

♦ 

0XL21 

EQU 

44, X 

DrL21 

EQU 

45, X 

DXL22 

EQO 

46, X 

DYL22 

EQO 

47, X 

* 

DXL31 

EQU 

48, X 

DYL31 

EQO 

49, X 

DXL32 

EQU 

50, X 

DYL32 

EQO 

51, X 

* 

KXL1 

EQO 

52, X 

KYL1 

EQO 

53, X 

* 

KXL2 

EQO 

54, X 

KYL2 

EQO 

55, X 

« 
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FILE: 

GPSSI 

VS  1  JOB 

KXL3 

EQU 

56, X 

KIL3 

EQO 

57, X 

BXL2 

EQO 

58, X 

EYL2 

4 

EQU 

59, X 

RXL3 

EQU 

60, X 

RyL3 

4 

EQO 

61, X 

CONVERSATIONAL  BONITOH  SISTEB 


**************  ************************************* 

*.  SERVICE  TIKES  OF  DEVICES,  BDSES,  PROCESSORS  * 

*  *  :ti  4c «  Itt  ^  :4t «  4i  4t  *  ^  ^  4c « t  *  4< 't' *  4<  *  «  *  *  1<  #  *  *  *  4>  4> «  *  >(•  *  4>  4t  *  *  4<  *  4<  4>  *  *  *  4c  *  4<  11 4i  *  <1  * 


BEX  11 

EQO 

62, X 

L(l)  STORAGE  SERVICE 

TIKE 

BEX  12 

EQO 

63, X 

DEX13 

EQO 

64, X 

DEX21 

EOU 

65, X 

L(2)  STORAGE  SERVICE 

TIKES 

DEX22 

EQO 

66, X 

DEX  31 

EQO 

67, X 

L(3)  STORAGE  SERVICE 

TIMES 

0EX32 

EQU 

68, X 

BEXDl 

EQO 

69, X 

BOS  SERV  TIKE  L  (1 J 

BEXI)2 

EQU 

70  ,X 

BUS  SERV.  TIKE  L  (2) 

BXD3 

EQO 

71, X 

BUS  SERV.  TIME  L  (3) 

BEXK 

EQU 

72, X 

BOS  SERV.  TIKE  FOR  KSG 

KEX 

EQU 

73,  X 

LEVEL  CONTROLLER  (K) 

SERVICE  TIME 

PEX 

EQU 

74, X 

HEHORX  REQUEST  PROCESSOR  (R)  SERVICE  TIKE 

TIHEE 

EQU 

75, X 

4i  4c  4i  41 4i  4t  4c  41  ir  4c  4<  4<  41 H  4>  4c «  41 4>  4>  4c  *  4  4i  4>  «  4<  4i  4> «  «  4i  4!  41 4c «  4i  H  4>  4i  4c  H  «  «  4<  4>  4c  4  *  *  4  «  4> «  4  41 4i  41 4>  4>  4c  4>  4i  4c  41 
4  4 

•  VAEAIBLE  DEFINITIONS  * 

4  4 

***************************************************************** 


KRESP  FVAEIABLE 
TXN«  VARIABLE 
TXNQ  VARIABLE 
TXNX  VARIABLE 
ETON  BVARIABLE 
BVA1  BVARIABLE 
BVA2  BVAEIABLE 
BVA3  BVARIABLE 
BVA21  BVARIABLE 
BVA«  BVARIABLE 
BVA5  BVAEIABLE 
BVA6  BVARIABLE 
BVA7  BVAEIABLE 
BVA8  BVARIABLE 
BVA22  BVARIABLE 
BVAV  BVARIABLE 
BVA10  BVARIABLE 
BVA11  BVARIABLE 
BVA12  BVARIABLE 
BVA13  BVARIABLE 
oVa23  BVARIABLE 


(X$SUflW/X$NTXN)  KEAN  RESPONSE  TIKE 

P3-P2  TXN  ELAPSED  TIKE 

P3-P2-P4  TXN  WAIT  TIKE 

P4 

(XiKXLl 'L' XSKXHI) ♦(X$KXL2'L' XJKXK2) *FH0$CB0S 

(X^DYLII'L'XSDYMn)  #FN0$DRP11 

(XSKXLI'L' XJKXMI) 4PNU$IBUS1 

(X$DYL21'L*X$DY!i21)  *FNIJ$DEP21 

(X$DyL22'  L*  XSDYM22)  4FNIJRDRP22 

(XiKXL2'L'X$KXM2) ♦FNU$LDUS2 

(X$KYL2'L*  X$KY«2) *FHOJKRP2 

(X$KXL1'L'X$KXK1) 4FNU$GDUS 

(X$DXL1 1 • L' XSDXKI 1) *FNU$IBUS1 

(XiDYLSI'L'XSDYKai)  4FNB$DEP31 

(Xi DYL32' L*  XSDYK32) ♦FNU$DRP32 

(X$KXL3' L*X$KXM3) ♦FNU$L3US3 

(X$KYL3*  L' XSKYK3) 4FNU$KHP3 

(XSRX12'  L'XiRX«2)  4FSU3J[,Bas2 

(X$RYL2'L' X$EYK2) 4FNUJRSP2 

(XJDXL21'L'X3DX«21) ♦FNaSLBUS2 

(X$DXL22' L*  X$DXn22) 4FNU$LBDS2 
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pile:  G?SS1 


VSIJOB  D^f 


CONVEBS&IIOMAL  EONITOB  STSIEH 


BV&14  BVABIABLE 
BVA15  BVABIABLE 
BVA16  BVABIABIE 
BVA17  BVABIABLE 
BVAia  BVABIABLE 
BVA19  BVABIABLE 
B7A24  BVABIABLE 
BVA20  BVABIABLE 


{XSKYLI'L'XSKYBI)  ’♦FNU3KUP1 
(X$KXL2'I,'  X$KXB2)  *FNO$G30S 
(X$KXL3'L'X$KXM3) *FHU$GBUS 
(X$P,XL3'L'  X$RXB3)  *FN0$LBUS3 
(X$KYi3'L»X$BXB3)*PNO$aRP3 
(XSDXL31 • L*  XiDXaSI) *FNO$LBOS3 
(X$DXL32'L'X$DXfi32)  *?N0$LBUS3 
(X$KYL1'L'X$KYB1)<‘PN0$KBP1 


tt,^*tminntt^m*********<ii*********************^^**  ******************* 
*  * 

♦  QTABLE  DEFINITIOMS  -  DISTHIBOTIOHS  OF  QUEUE  LENGTHS  • 

•  ■  •  * 

1c««**«*«*«*4i4i«*«i****  «««***««««** 


*  * 

♦  PONCTIOH  DEFINITIONS  • 

*  * 
***************************************************************** 

HZCHH  FUNCTION  F1,D3 
2,HIIN11/3«VHU12/4,«WW13 

ItlCHA  FUNCTION  P1,D3 
2,AAA1V3,AAA12/4,AAA13 


*«***«*4i««*  ***«*«  **««««****  «*****«**«««***« 


*  * 

♦  TABLE  DEFINITIONS  -  DISTBIBUTIONS  OP  TXH  ELAPSED  TIHE,  * 

♦  WAIT  TIBE  • 

♦  * 


***************************************************************** 


TXNW  TABLE 
TXKQ  TABLE 
TXMX  TABLE 


V$TXN«« 100,100, 100 
VSTXNQ, 100,100,  100 
VSTXNX, 100,100,100 


***************************************************************** 
*  * 

•  INITIALIZE  CONSTANTS  ♦ 

*  * 
***************************************************************** 


INITIAL  XSBAXNP,20 

INITIAL  X$HREAD,7,00 

INITIAL  X$NWEIT,300 

INITIAL  X$PIB1,400 

INITIAL  X$PIN2,400 

INITIAL  X$PIN3,1000 

INITIAL  X$PCV1,500 

INITIAL  X$POV2,500 

INITIAL  X$DXB11,10 


DEGREE  OF  BULTIPBOGBABBING  OF  A  CPU 
%  BEAD  TXN 
%  WRITE  TXH 

PEOB  OF  FINDING  BEAD  DATA  IN  L(1) 
PROS  OF  HOT  IK  L(1)  AND  IN  L(2) 

PBCB  OF  FINDING  DATA  IN  L(3) 

PROB  OF  OVERFLOW  FEOB  L{1) 

PROB  OF  OVERFLOW  PBOH  L(2) 

BAXIBUB  DATA  QUEUE  LENGTH 
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INITIAL 

Xt BY  Hi  1,10 

INITIAL 

X$DXM12, 10 

INITIAL 

XlDYtl12,  10 

INITIAL 

X$DXM13, 10 

INITIAL 

X$DYfi13,  10 

INITIAL 

XSDXH21, 10 

INITIAL 

x$Dyn2i, 10 

INITIAL 

X$DXM22, 10 

INITIAL 

X$DYn22,10 

INITIAL 

X$DX(131, 10 

INITIAL 

X$CYK31, 10 

INITIAL 

X$DX«32, 10 

INITIAL 

XJ DYM32, 10' 

INITIAL 

X$KXK1,  10 

INITIAL 

XSKYWI,  10 

INITIAL 

XSKXH2, 10 

INITIAL 

X$KY«2, 10 

INITIAL 

X$KXn3, 10 

INITIAL 

X$KyM3, 10 

INITIAL 

XSRXM2, 10 

INITIAL 

X$RYfl2,  10 

INITIAL 

X$RXB3, 10 

INITIAL 

XARYB3, 10 

INITIAL 

X$DEX11,100 

ACCESS  TIME  OF  Dll  IN  NAHOSEC 

INITIAL 

X$DEX12, 100 

INITIAL 

X$DEX13,100  - 

• 

INITIAL 

X.<;DEX21, 1000 

ACCESS  TIME  OF  D21  IN  NANOS3C 

INITIAL 

X?DLX22, 1000 

INITIAL 

X$DKX31, 10000 

ACCESS  TIME  OF  D31  IN  NANOSEC 

INITIAL 

X$DEX32, IOCOO 

INITIAL 

XABKXDI, 100 

BUS  SERV.  TIME  IN  NANOSEC 

INITIAL 

XSBEXD2, 1600 

INITIAL 

X$nEXM,  100 

INITIAL 

X$KEX,100 

L(I)  CONTE.  P.  SERV.  TIME  IN  NAKOS 

INITIAL 

XSREX,200 

HEQ.  P.  SERVICE  TIME  IK  NANOS 

INITIAL 

XSTIMEE, 1300000 

SIMULATION  TIME 

* 

« 

*  MACRO  -UTX 

« 

UTX  STARTMACRO 

SEIZE 

tA 

DEPART 

*B 

ASSIGN 

4»,#C 

ASSIGN 

7,»C 

ADVANCE 

P7 

RELEASE 

«A 

ENDMACBO 
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PILE:  GPSSI 


VSIJOB 


CONVERSATIONAL  MONITOR  STSTBM 


* 

•  MACRO  -  UQTQ 

* 


* 

* 

* 


DQTQ  STARTHACRO 


QUEUE  #A 

SEIZE  «B 

DEPART  #A 

ASSIGN  4*,#D 

ASSIGN  .  7,#D 

ADVANCE  P7 

RELEASE  #B 

QUEUE  «C 

ENDMACRO 

4>  <1 4<  4>  *  *  *  *  *  ^  %  4i  *  <1  *  4: 

♦  ♦ 

♦  MACRO  -  UQT  * 

*  * 


******  ********  ****** 


STARTMACRO 

QUEUE 

»A 

SEIZE 

#B 

DEPART 

♦A 

ASSIGN 

4  + 

ASSIGN 

7. 

ADVANCE 

P7 

RELEASE 

IB 

ENDMACRO 

********************* 

* 

*  MACRO  -  UQDQ 

* 

********************* 


************** 

* 

* 

* 

************** 


STARTSACEO 

QUEUE 

#A 

TEST  E 

#G,1 

S AVEVALUE 

tD>1 

SEIZE 

IE 

DEPART 

tA 

S AVEVALUE 

#D,1 

ASSIGN 

4+,#P 

ASSIGN 

7,#P 

ADVANCE 

P7 

RELEASE 

AE 

QUEUE 

#C 

ENDMACRO 

************************* 


yiLE:  GPSSI 


VS1J0B  D7 


CONVERSATIONAL  MONITOR  STSTEM 


MACRO  -  UQD 


* 

*  * 
******  ******1^'******* 


STABTMACEO 

QUEU  E 

#A 

TEST  E 

tG,1 

S AVEVALUE 

SD,  1 

SEIZE 

#E 

DEPART 

#A 

SAVEVALUE 

#B,  1 

ASSIGN 

N+,#F 

ASSIGN 

7,#F 

ADVANCE 

P7 

RELEASE 

iE 

ENDKACRO 

*********************************** 
*  * 

*  MACRO  -  PINI  * 

♦  ♦ 
*********************************** 


FINI 


STARTHACKO 

MARK 

S AVEVALUS 
S  AVEVALUE 
S AVEVALUB 
S AVEVALUE 
S AVEVALUE 
TABULATE 
TABULATE 
TABULATE 
ASSIGN 
ASSIGN 
.  ASSIGN 
ASSIGN 
ASSIGN 
ASSIGN 
ENDHACRO 


NTXN  +  ,  1 
SU  HX  +  ,VSTXNX 
SUMQ+, VITXNQ 
SUHW+,V$TXNM 
MEESP, ViMRSSP 
TXNW 
TXNQ 
TXNX 
1,0 
2,0 
3,0 
4,0 
5,0 
6,0 


SIMULATE 

*********************************** 
*  * 

♦  CPU  #1  * 

*  * 
*********************************** 


CPU1  GENERATE  , , , X$KA XMP, , , F 
************************* 

*  START  FOR  CPU1  TXNS  * 

START  PRIORITV  9 
MARK  2 

ASSIGN  1,1 


SET  HIGH  PEIORITX 
ARRIVAL  TIME  OF  TXN 
SET  CPU  ID  =  1 
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FILE:  GPSS1 


TS1J0B  D? 


COHVEBSATIONAL  BOMITOB  SISTBS 


TRANSFER  .  X$NR£AD,KUU1  ,RflR1  BEAD  OR  HKXTE  TXN? 

* ««*««««**«  4r  4ci|i 


♦  READ  TXN  PROM  CPU1  • 

1^  **  ««««««« t  «*««  4> «  «««*  « 


BBB1  QUEUE 

DIQ11 

SEIZE 

DEP1 1 

DEPART 

D1Q11 

PRIORITY 

0 

ASSIGN 

4+,X$REX 

ASSIGN 

7,X$EEX 

ADVANCE 

P7 

RELEASE 

DRP1 1 

TRANSFER 

.X$FIN1 ,N1N1 

BEAD  TXH 

RESET  PRIORITY 

TIHE  FOB  DIRECTORY  SEARCH 

flNDII  IS  DATA  IN  L(1)7 


•  BEAD  TXN  FROn  CPU1  * 

♦  IS  SATISFIED  IN  L{1)  * 

****  DC  I|t4i  :|e  ««*«*««*  ««*«««  ic  *  * 


IND11  ASSIGN  11,0 

t*********************i^tt 

*  BEAD  DATA  FROM  Dll  * 

^ttHit******************** 

UQT  MACRO  DIQn«DBPl1,X$0£X‘l1 

************** 

*  USE  FXKI  MACRO  * 

*  THE  TXN  IS  COMPIETED  • 

************************* 

FIHI  MACSO 

TRANSFER  ,STAR1  THE  TXN  BECOMES  A  NEW  TXN 

************************* 

*  BEAD  TXN  FROM  CPB1  IS  ♦ 

*  NOT  SATISFIED  IN  L(1)  * 

************************* 


NIN11  QUEUE  DOQ11 


************************* 


*  USE  UTX  TO  USE  * 

*  THE  LOCAL  BUS  LBUSl  * 

************************* 

DTX  MACRO  LDDS1,DOQ11,X$BEXH 

TRANSFER  ,CO«R  GO  TO  COMMON  CODE  FOB  BEAD 

************************* 


*  WRITE  TXH  FROM  CPU  1  * 

************************* 


VWWI  QUEUE 
TEST  E 
SAVEVALUE 
SEIZE 
PBIORITY 
DEPART 


DIQ11 
BVSBVAI, 1 
DYL11+,1 
DRPII 
0 

DIQII 


Dll  OUT  QUEUE  AND  OBP  FBEE? 
SAVE  SPACE  IN  OUT  Q 

RESET  PBIOBITY 


FILE:  GPSSi 


VS1J0B 


O'! 


COHVEESATIOSAL  MONITOR  SXSTEH 


ASSIGN  <»+,X$DEX11 

ASSIGN  7,X$DEX11 

ADVANCE  P7 

RELEASE  DHP11 

SPLIT  1,STB1 

t*****t‘***********'ir****** 

*  WRITE  TXN  IS  COMPLETED* 

FINI  MACRO 

TRANSFER  , STAR  1 

4>  *  *  *  « <<>  4c  «  4c  1C  :|c  «  *  *  «  *  4i  *  «  «  « 

*  STORE- BEHIND  TXN  * 

*4c4c4titi  4riic4c4c4c4[4i4c!ti4i4i4c4c4l4i4c4ciici|ci)i 

STB1  QBEUE  DYQII 


TEST  E 
SAVEVALUE 


BV$BVA2,1 

KXLU,1 


TIEE  FOB  WRITING  DATA 


CREATE  A  STOBE-BEBIND  TXN 


BECOMES  A  HEW  TXN  FROM  CPUI 


PDT  TXN  IN  DATA  QOEDE 

K1  IN-Q  AND  LBUS1  FREE  ?  ’ 
RESERVE  SPACE  IN  IN-Q 


***********¥******** 

*  OSE  LDOS1  TO  SEND  TXN  * 

•  FPOH  Dll  TO  K1  * 

^^*t***if****************** 

HTX  MACRO  L3US1,DyQl1,X$BEXD1 


SAVEVALUE 

TRANSFER 


DYL1 1-,  1 

,cOaw 


RELEASE  SPACE  IN  Dll 
TO  COMMON  CODE  FOE  WRITE 


«  «  *  «  4c  *  4c «  4<  4>  *: «  4c  4c  4c  4c  4c  c|c  4c  4c  4> «  4c  4c  4c 

*  COMMON  CODE  FOE  * 

*  READ  TO  LOWER  LEVELS  * 

*  JOINED  BY  ALL  CPUS  * 

*  4c  4c  41  ***  4c  *  4c  *  4c  4c  4c  4c  *********  * 


COME  ASSIGN  11,0 

***«  4c  4c  4c  4c  4c  4c  4c  4c  4c  *********  * 

*  USE  K1  * 

*  «  4c  *  4c  4c  4c  4c  4c  4c  4c  *  4c  4>  4c  4C  *  4‘ 4c  4c  *  4c  4<  41 4c 

DQTQ  MACRO  K IQ1 , KRP 1 , KOQ1 , X$KBX 

*  4c  4c  4c  4!  *  4c  4c  *  4' 4c  4c  4c  4c  4C  4c  4c  4c  4!  4c  4c  r  4c  4c  * 

*  USE  GLOBAL  BOS  GBUS  * 

4c  4c  4c  4c  4c  4c  4>  *  *  4i  4c  4c  4c  4c  4c  4i  *  *  *  *  *  *  *  *  * 

OTX  MACRO  GBUS,KOQ1,X$BEXfl 

4c  4!  4c  4c  4c  4‘4c  4c  4c  4c  4c  4c  4c  4c  4' 4c  4c  4c  4c  4c  4c  4c  4c  4c « 

*  OSE  K2  * 

4t «  4c  4c  41  *  4c  4c  4c  4c  4c  4c  4c  41 4c  4>  4c  4c  4c  4c  4c  4c  4c  4c  4c 

UQTO  MACRO  KIQ2  ,  Kr<P2 ,  KOQ2,  X$KEX 

*  4>  4c  *  4c  *  4>  *  4c  4c  41  4c  41 4c  4!  41 4c  4c  4C  41 4c  4c  4c  4c  4c 

*  OSE  LOCAL  BUS  LDnS2  * 

«  4c  4c  4c  41  **  *  41 4c  4c  41  *  *  4i  41 41 4i  4c  41 4c  4c  41  *  41 

OTX  MACRO  LBUS2,KOQ2,X$BEXM 

*  41 41 4c  41  *  *  *  *  4C  *  *  *  4<  4>  41 41  *  4c  4c  4>  4c  4  41 4c 

*  OSE  R2  TO  SEE  IF  DATA  * 

*  IS  IN  L{2)  * 

4i  4i  4i  4>  *  *4  4  4i  4  4i  4>  4i  4c  4c  *  *  «  *  *  *  *  *  *  * 


DUMMY  STATEMENT 


DQT  MACRO 


RXQ2,RRP2,XSR£X 


FILE:  GPSS1 


VSIJOB  D<o 


CONVERSATION &L  KONITOR  STSIEM 


TRANSFER  . X$FIN2,NIN2, INL2  IS  DATA  IN  L(2)7 

<1*  *  <1 4141 « lit «  ^  4<  41 «  « 1 4I)||  41 

*  DATA  IS  NOT  FOUND  IN  • 

*  L(2)  * 

NIN2  QUEUE  BOQ2 

4t  4t  4[i|i  4t  41 411I1 «  41 « It  4t  4t  41 «  4i  41 4c  41 41 « lit  4i  4c 

*  USE  LBOS2  SEND  TIN  TO  • 

*  K2  * 

*  «  41 41 4i «  4>  4>  4t «  4i  41 4i  4i  4i  4i  4>  4<  41 4t  41  *  41 4i  41 

OTX  NACRO  LDUS2,BOQ2;XSBEXH 

*  4t  41 4t  4>  4>  4>  4<  4' 1 41 4>  4i  *  4>  41 4>  4: 41 4t  4i4c  4>  4>  * 

*  SERVICED  BY  K2  * 

*  4i*  41 4>  41 4i  4<  4c  41 4i  4c  41 4<  41 4<  4t  4>  4<  4>  41 4t  4<  4i  41 

icQTQ  HACRO  XIQ2 ,  KRP2,  KOQ2.  XSKEX 

41 4c  4t  4c  4c  *  4>  4t  4c  4e  4t  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4>  4c  4c 

*  USE  GBUS  SEND  TXN  TO  * 

*  K3  * 

«4i  41 4c  4>  *  4c  4>  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  41 4c  4c  4c  4c  4c 

OTX  HACRO  GD0S,K0Q2,X$DEXH 

4c  *  4c  it  *  4c  4c  4c  4>  4t  4c  ♦  *  4c  4c  4c  4<  4c  *  4c  4c4' 4c  4c  4c 

*  SERVICED  BY  K3  * 

41  *4' 4c  4>  41  **  4i  4c  4c  4c  4c  4c  *  4c  4c  4c  4c  4c  4c  «*  4c  4c 

UQTQ  HACRO  KIQ3 , KRP3 ,KOQ3,X$KEX 

4i«  41 4c  *  4c4c  %  4c  4c  4c «  4c  4c  4i  4c  4c  41 4c  41 4c4c  4c  4c4c 

*  USE  LBOS3  SEND  TXN  TO  ♦ 

4c  83  4c  . 

4t4c  4c  4c  4c  4c  4c  4>  4c  4c  4>  4c  4c  4c  4t  4c  4' 4c  4c  4c  4c  41 4c  It  4c 

OTX  HACRO  LnUS3,KOQ3,X$BEXH 

4c  4c  4c  *  4c  4c  41 4c  4c  4c  It  4c ««  4c  4c  4c  4c  4c  4c  4c  It  4c  4c  4c 

*  SEARCH  DIRECTORY  IN  * 

*  R3  FOR  DATA  * 

41  *  *  4c  4c  4c4c  4c  4c  4c  4c  4  41 4c  4c  4c  *  4c  *  4c  4c  4c «  *  4> 

UQT  MACRO  RIQ3,RRP3,X$REX 

TRANSFER  ,INL3  DATA  IS  IN  L(3) 

ct  4l  4c  4c  4c  4c  4c  4c  4c  4>  4c  4c  4c  4c  4C  4c4c  4c  41 4c  4c  4c  4c  4c  4c  4C  4c  4c  4>  4c  4  4  4c  4c  4c 

4C  data  is  found  in  L(2),  READ  THE  ♦ 

*  DATA  AND  SEND  IT  DP  TO  L  (1)  ♦ 

44444444444444444444444444444444444 

1NL2  QUEUE  ROQ2 

4444444444444444444444444 

*  send  TXN  TO  DEVICE  • 

*  VIA  LDUS2  * 

4444444444444444444444444 

DTX  HACRO  LBUS2,ROQ2,X$B£XH 

*444ct  444444  444  444444  44  444444444444 

*  IS  DATA  IN  Dll  OH  D12?  • 

4*4«4444444  44444444444444444ct44444 
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FILE:  GPSSl 


VSUOB  Dll 


CONVEBS&TICHAL  SOSITOB  SFSTSH 


TRAHSFER  .5,PBB21,I!Ilfi22 

*  DATA  IS  IN  Dll  ♦ 

«4i  4141 *«*««««  ««*««« 


ERE21  QUEUE  DIQ21 

TEST  E  BVJDVA3,1 

SAVEVALOE  DyL21+,1 


QUEUE  TO  EETKIEVE  DATA 
D21  OUT-Q  AND  DRP21  FREE? 
SAVE  SPACE  IN  D21  OOT-Q 


41 4  *  4  4  *  4  4  4  4c  4t  4  4  4  4  4  4  4  4  4  41 4  4  4  4 

*  USE  D21  TO  RETRIEVE  * 

♦  THE  DATA  * 

4444444444*44444444444444 

DTX  SACRO  DSP21,DIQ21 

QUEUE  DiQ21 


X$DEX21  RETRIEVE  THE  DATA 
PUT  DATA  IN  SLOT 


TEST  E  BV$DVA4,I 

SAVEVALUE  KXL2*,1 


K2  IH-Q  AND  LBOS?  FREE? 
RESERVE  K2  IN-Q  SLOT 


4444444444444444444444444 

*  USE  LBUS2  SEND  DATA  TO  * 

*  K2  * 

4444444444444444444444444 

UTX  MACRO  LBUS2,DYQ21,X$BEXD1 


SAVEVALUE  DYL21-,1  RELEASE  SLOT  IN  D21  OUT-QUEUE 

TRANSFER  ,RTF2  TO  CODE  FOR  READ-THROUGH  FROM  L (2) 

44444444444444444444 

♦  DATA  IS  IN  D22  • 

44444444444444444444 


BRE22 

QUEUE 

DIQ22 

TEST  E 

BV$BVA21, 1 

SAVEVALUE 

DIL224,  1 

UTX 

MACHO 

DRP22,DIQ22,X$DEX22 

QUEUE 

DXQ22 

TEST  E 

DV$3VA4, 1 

SAVEVALUE 

KXL2  +  ,  1 

UTX 

MACRO 

LBUS2, 0X022, XSBEXDI 

SAVEVALUE 

0X122-, 1 

TRANSFER 

,BTF2 

4444444444444444444444444444444*444 

♦BEAD  THROUGH  FROM  LEVEL  L(2)  * 

*4*4444444444444444444444444444*4*4 

BTF2  ASSIGN  11,0 
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file:  gpssi 


VS1J0B  DI2 


CONVEBSATIONAI  80NIT0B  SXSTEH 


♦  SERVICED  BV.  K2  • 

OQDQ  MACRO  KXQ2,KXL2-,K1Q2,KYL2+,KBP2,X$KEX«BV$BYA5 

TEST  E  BV$DVA6, 1  K1  IH-Q  AMD  GBOS  FREE? 

savevalde  kxlu,i  reserve  ki  im-q  slot 

•  OSE  GBUS  TO  SEHD  DATA  TO* 

•  KI  * 

OTX  MACRO  GBUS,KVQ2«XSBEXD1 

SAVEVALnS  KTL2-,1I  RELEASE  SLOT  IH  K2 

*«i|l4c««i|c*4!t«*>)'***««***it'************* 

*  STORE  DATA  INTO  L(1)  AS  A  RESULT* 

•  OF  HEAD-THROUGH  * 

^t^itit^tt**************  ************* 

STOR1  ASSIGN  11,0 

************************* 

*  SERVICED  DV  KI  * 

************************* 

UQD  MACRO  KXQ1,KXL1-,,KTLU,KBP1,X$KEX,BV$BVA20 


********************************* 
*  SEND  TO  Dll  OH  D12  • 

********************************* 


SPLIT  1,PN$WICHH,1 

.  TERMINATE 

******************** 

*  STORE  TO  B11  * 

******************** 


BVH11  ASSIGN 
QUEUE 
TEST  E 
SAVEVALUE 


11,0 

KVQ1 

BVSBVAT, 1 
DXL11+, 1 


•  SEND  TXN  TO  Dll  VIA  • 

•  LBUS1  • 

•  «*«**««* 


HKICH  DATA  CACHE  TO  GO? 


HRITB  TO  Dll 

SPACE  IN  Dll  IN-Q  AND  LBDSI  FREE? 
TES,  RESERVE  A  SLOT 


OTX  MACRO  LBDS1,KTQ1,X$BEXD1 

SAVEVALUE  KVL1-,1  RELEASE  KI  SLOT 

************************* 
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FILE:  GPSS1 


VS1J0D  di3 


CONTESSmONAL  EOtllTOS  SISIEH 


*  WRITE  DAT^  TO  Dll  * 

iti  *  41 4i  t  ^  f  4<  t *  4' «  «  *  *  *  *  <*<  t  <t>  * 

UQT  MACRO  DXQ11,DHP11,X$DEX11 

SAVEVALUE  DXL11-,1 

TRAWSFER  . XSPOV 1 ,HOV 1 1,OV11 1  AHY  OVERFLOW  FROM  L{1)? 

*  NO  OVERFLOW  FROB  L(1)  * 

*  «  4[ «  *  *  4>  4<  *  4>  4>  4i  *  41 «  *  *  *  *  4< «  «  *  * 

NOV  11  ASSIGN  11,0 

* *  41 «  «  «  41 4E  4<  <)<  4<  *  *  4e  *  *  *  *  «  4i  *  * 

*  THE  READ  TXN  HAS  ENDED* 

« 1 41  •  4>  *  «  *  4i  4<  4i  «  *  *  41 4>  41 4i  41 41  *  4i «  «  * 

riHI  MACRO 

TRANSFER  ^STARl 

4  41  4e  41 4  41 41 4r  4t  4>  41  41 4i  4<  41 4  4  4r  4t  41  4  4<  4>  4>  41 

*  THERE  IS  OVERFLOW  FRCB* 

*  L  {1) ,  END  THE  READ  * 

*  TXN,  AT  THE  SAME  TIME  * 

*  HANDLE  THE  OVERFLOW  * 

*  4  4  4i  4  4  4r « -ii  41 1' 4  41 41 4>  *  4<  4>  4  4>  41 4<  41 41 4> 

OVL11  SPLIT  1,0¥F11  GOT.  OVERFLOW  HANDLING 

FINI  MACRO  AT  T  'E  SAME  TIME  END  THE  TXN 

TRANSFER  ,STAR1 

44>4i  44<4144<4444<4<  44  4c4«4444'44>4 

*  OVERFLOW  HANDLING  FOR  ♦ 

4  Dll  * 

4  4444  44444  4<4<44444444>444>4»k 

OVF11  ASSIGN  11,0 

DOT  MACRO  DOQ11,LBUS1,XSB£XH 

TRANSFER  ,OVL1  GOTO  COMMON  CODE  FOB  OVERFLOW 

4444444444444444444444444444444444444 
4  WHW12  * 

4444444444444444444444444444444444444 

44444444444444444444444444*4444444444 

4  WWW13  • 

4444444444444444444444444444444444444 

IIWW12  ASSIGN  11,0 

WHH13  ASSIGN  11,0 

4i*«««4444  44444  444444  44  44  444  44444444 

*  COMMON  CODE  FOR  OVERFLOW  FROM  * 

*  L(1)  • 
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PILE:  GPSS1 


VS1J0B  DI4 


COMVERSfcTIONAL  HOHZTOR  SYSTEH 


0VL1 

ASSIGN 

11, C 

**************************************** 

*  USE  K1,  THEM  GBUS,  THEM  K2  * 

•  THEN  LBnS2,  THEN  USE  B2 

**************************************** 

OQTQ 

HACBO 

KIQ1,KRP1,K0Q1,X$KEX 

UTX 

HACBO 

GBDS,KOQ1,X$B£XH 

DQTQ 

HACBO 

KI Q2 , KB  P2 , KOQ2, X$KEX 

DTX 

HACBO 

LBUS2,KOQ2,X$BEXH 

QUEUE 

BIQ2 

DTX 

HACBO 

BRP2,fiIQ2,X$BEX 

TERMINATE 

******«****4>4>*«*«i|i«*«*«4i4> 

*  DATA  IS  FOUND  IH  L(3)  * 

*;^tHi*4f*0******<im******i**0 

1NL3  QUEUE  ROQ3 

Iflft********  ******  ******** 

*  USE  LBOS3  SEND  TXN  TO  * 

*  D31  * 

**«*# ******************** 

DTX  HACBO  LBDS3,BOQ3,X$B£XN 


***************************** 

♦  READ  FROM  D3 1  OR  D32?  ♦ 

TRANSFER  .5,F.RR31  ,RRR32 

******************** 

*  READ  FROH  D31  * 

******************** 

BER31  QUEUE  DIQ31 

TEST  £  BY$BVA8,1  SPACE  IN  D31  OUT-Q  AND  DBP31  FREE? 

SAVEYAEUE  DYL31+,1 

*  *  4i«  4  *  *  *  *  4>  4t  4i  Ik  *  «  «  *  >l<  4<  <>  4>  4>  4 

•  BEAD  DATA  FROB  D31  • 

************************* 


263 


FILE:  GPSS1 


VSIJOB  015” 


CONVERSATIONAL  HOKlTOfi  SVSTEH 


OTX  MACRO  DBP31,DIQ31,X$DEX31 

QUEDE  DVoai 

TEST  E  BV$DVA9,1  SPACE  IN  K3  IN-Q  AND  LBUS3  FREE? 

SAVEVALOE  KXL3*,1  YES,  RESERVE  SLOT 

4i4ii|t «  «  « « 4c  <t<  *  >ic «  4^  4< « >tc  *  «  #  *  «  «  * 

*  OSE  LB0S3  SEND  DATA  TO  * 

♦  K3  * 

OTX  MACRO  LBUS3,DYQ31,X$BEXD2 

SAVEVALOE  DYLSI-,! 

TRANSFER  ,RTF3  GO  TO  READ-THROUGH  FROM  L(3) 

y  %  i)c  4c «  «  «  V  «  « ic « 41  * 

*  READ  FROM  D32  ♦ 

«  41 4c «  «  *  «  4t  4c  t  itc  4c  4c  4t  4c «  4> «  «  « 

RBB32  QUEUE  D1Q32 

TEST  E  BV$BVA22,1 
SAVEVALOE  DYL32+, 1 

OTX  MACRO  DRP32,DIQ32,X$DEX32 

QOEHE  DYQ32 

TEST  E  BV$IiVA9,1 
SAVEVALOE  KXL3+, 1 

OTX  MACRO  LB0S3,DYQ32,X$BEX02 

SAVEVALOE  DYL32-,1 
TRANSFER  ,ETF3 

41 4c  4c  4c  41 « 4>  4>  *  4c  4c  4c  4c  4c  4> «  4c  4c  4<  4 1  *  4c  4<  4c  4>  4c  4<  4c  4c  4c  *  «  4<  4c 

♦  READ-THROOGH  FROM  L{3)  DATA  IS  * 

•  SENT  TO  L{2)  AND  L(1)  AT  THE  ♦ 

•  SAME  TIME  * 

4c  4  4c  4c  4i  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4>  4' 4>  4c  4c  4c  Ik  4  4c  4c  4c  4<  4c  4  4c  >4  4<  4>  *  * 

RTP3  ASSIGN  11,0 

444*c4444  444  4  4c4  4  44«44*«4c4« 

*  SERVICED  BY  K3  * 

444  itc4  *  4c|c  44  4  *  4  44  c4  44  >4  cfc  44  ctc4* 

OQDQ  MACRO  KXQ3,  KXL3-,KYQ3,,KYL3*,KRP3,X$KEX,B7$BVA10 

TEST  E  BV$RT0K,1  L{1)  &  L  (2)  READY  6  GBOS  FREE? 

SAVEVALOE  KXL1+,1 
SAVEVALOE  KXL2+,1 

4444444444444444444**4444 

•  BOTH  L(1)  AND  L  (2)  ♦ 
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FILE:  6PSS1 


VS  1  JOB  Dl(f 


COtlVERS&XlOHAL  MONITOR  SYSTEM 


•  BEADY  TO  ACCEPT  DATA  *  ' 

•  PROM  GBUS  * 

UTX  MACRO  GDUS«KYQ3«X$B£XD2 

SAVEVALOE  KYL3-,1 

SPLIT  1,ST0R1  READ-THROOGH  TO  L(1) 

•  **4*««4>***«***«******4>«* 

•  READ-THROUGH  TO  L  (2)  « 

STOB2  ASSIGN  11,0 

««*«***«*«4>****4<»*** 

•  SERVICED  BY  K2  * 

M*********************** 

OQDO  MACRO  KXQ2,KXL2-,KYg2,KYL2«,KRP2,XSKEX,DV$BVA5 

TEST  E  BV$BVA11,1  SPACE  IN  R2  IN-Q  AND  LB0S2  FREE? 

SAVEVALUE  RXL2«,1  YES,  RESERVE  SLOT 

•  USE  LDns2  SEND  TO  B2  * 

M*********************** 

OTZ  MACRO  LBUS2,KYQ2,X$DEXD2 

SAVEVALUE  KYL2-,1  FREE  SLOT  IK  K2 

•  SERVICED  DY  R2  ♦ 

DQD  MACRO  BXQ2, RXL2-, , BYL2«,RRP2, X$BEX, BV$BVA12 

SPLIT  1,OVH2  HANDLE  ANY  OVEBFLON 

•  STORE  INTO  D21  OR  022?  * 

****4i««««i««***«4i*4i««i|i«**W**«*«« 

TRANSFER  .5,SSS21,SSS22 

*4i4>  *41  *«««««*«**«*« 

•  STORE  INTO  D21  • 

*««•«««****«««*«*«*»***«« 

SSS21  QUEUE  RYQ2 

TEST  E  BV$BVA13,1  D21  IN-Q  AND  LBDS2  FREE? 

SAVEVALUE  0X121+, 1  YES,  RESERVE  THE  SPACE 

Ai***************************** 
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FIIE;  GPSS1  VSUOB  Dir  COMV HHSATIOHAL  MOHITOB  SfSTUS 


♦  SEND  DATA  TO  D2 1  VIA  DUS  ♦ 

OXX  MACRO  LBUS2,RY02,X$BEXD2 

SAVEVAl.UE  Hn.2-,1  RELEASE  SPACE  IN  R2 

UQ7  MACRO  DXQ21,DaP21,X$DEX21 

SAVBVALUE  T)XL21-,1 
TERMINATE 

*■  STORE  INTO  D22  ♦ 


SSS22 

OUEUr, 

TEST  E 
SAVBVALUE 

RyQ2 

I)V$BVA23.  1 

DXL22+,  1 

OTX 

MACRO 

LBUS2,HYQ2,X$BEXD2 

S  AVEVALUE 

RyL2-, 1 

DOT 

MACRO 

DXQ22,DaP22,X$DEX22 

SAVEVALUE 

TERMINATE 

DXL22“, 1 

If  ***■¥*  *****t***^’t>** 

*  HAND.  ANY  OVERF.  FROM  1.(2)  * 

OV!l2 

OVL2 

TRANSFER 

QUEUE 

. XAPOV2, NOV2,OVL2 
ROQ2 

*****<|t**#***#**»*»>**  +  *4i<>1< 

*  OSE  LBUS2,  USE  K2,  USB  ♦ 

*  GBOS,  USE  K3.  USE  LDUS3,  « 

*  THEN  USE  R3  ♦“ 

«  4: 1)1  #  «  «  *  «  «: «  «  «  «  4=  *  4>  «  « 't' *  « 

DTX 

MACRO 

IBUS2,R0Q2,X$DEXM 

HQTQ 

MACRO 

KIQ2,KRP2,KOQ2,X$KEX 

OTX 

MACRO 

GDU.S  ,KOQ2,X$BF.XM 

OQTQ 

MACRO 

KIQ3,KRP3,K0Q3,X$KEX 

OTX 

MACRO 

LBUS3,KOQ3,XiBEXa 

UQT 

MACRO 

EIQ3,RRP3,X$REX 

HOV2 

TERMINATE 
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PILE':  6PSS1 


fSlJOB  D/p 


COKYEESATIONAL  EOVXOIOB  SPSTEtt 


•  COBBCN  CODS  FOR  WRITE  ♦ 

♦  TO  LOWER  LEVELS  * 

*t:*********************** 

COHW  ASSIGR  11,C  OffHHr  STATEMENT 

********4i4>**«*4>'t>******«** 

♦  SERVICED  BY  K1  ♦ 

*«*4i*«**i»««4i**«**4i*«*4i*** 

OQDQ  MACRO  KXQ1 ,KXL1-,KYQl,KTL U,KRP1,X$KEX,BV$BVA14 

TEST  E  BVSBVA15,1  K2  IN-Q  AND  GBDS  FREE? 

SAVEVALOE  KXL2+,1 

*****  **>|i  *«*«**  K********** 

*  OSE  GBOS  • 

************************* 

OTX  MACRO  6BUS,KYQ1,X$BEXD1 

SAVEVALOE  KYL1-,1 

************************* 

•  SERVICED  BY  K2  * 

************************* 

OQDQ  MACRO  KXQ2,KXL2-,KYQ2«KYL2«,KRP2,X$KEX,BVSBVA5 

TEST  E  BV$BVA11,1  B2  IN-Q  AND  LBUS2  FREE? 

SAVEVALOE  RXL2^,1 

************************* 

*  OSE  LB0S2  • 

************************* 

OTX  MACRO  LBOS2, KYQ2,X$B£XD1 

SAVEVALOE  KYL2-,1 

************************* 

•  SERVICED  BY  R2  * 

************************* 

OQD  MACRO  BXQ2,RXL2<-,,RYL2*,RBP2.XSREX,BVSBVA12 

*****************************ii‘*** 

*  SERVED  BY  D21  OR  D22?  * 

********************************* 

TRANSFER  . 5,SNS21 ,SRS22 


************************* 
*  SERVICED  BY  D21  * 

************************* 
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CONVEBS&TIOHAL  aOHITOR  SYSTEM 


SHS21  QUEUE  BXQ2 

TEST  E  BV$BVA13,1 
SSVEVALUE  DXL21+,1 

OTX  MACRO  LBUS2,RTQ2,XSB£XD1 

SAVEVALUE  RYI,2-,1 

UQDQ  MACRO  DXQ2 1 ,DXL2 DYQ2 1, 0YL21 +, DRP21 , X$DEX2 1 , BV$BVA3 

TEST  E  BV$BVA4,1  K2  IH-Q  AMD  iBUS2  FREE? 

SAVEVALUE  KXL2*,1 

*  *  <c  4<  *  *  <1  *  *4c  Iti  *  it «  «  4I  #  Ifr  * 

*  USE  LD0S2  SEND  TO  K2  * 

Ik  ^  *  it «  *  *  it  it  it «  «  «  ii>r  it «  it  it  *  it  it  it 

DTX  MACRO  LBUS2,DYQ2Y,X$B£XD2 

SAVEVALUE  0X121-,! 

SPLIT  1,ACK2  PREPARE  TO  SEND  ACK  TO  L(1} 

TRANSFER  ,STD23  CO  TO  STORE-BEHIND  TO  L{3) 

*  it  *  « *  ink  «  «  «  it « it «  it  k  «  it  it  it  k  4  it  *  *  it «  it  it « 

*  SEND  ACK  TO  L{1)  * 

it  *  *  it  it «  *  it  it  it  it  it  it  it  it «  it  *  «  it  it  ic «  k  «  k  «  «  «  it 

ACK2  QUEUE  D0Q21 

DTX  MACRO  LBUS2,D0Q21,X$B£XH 

TRANSFER  ,ACK21 

*  *  «  k  k  k  k  k  k  k  *  k  it  k  k  k  *  it  it  it «  k  k  k  k  it  k  «  « 

*  SERVICED  BY  D22  * 

kkkiikkkkkkkkkkkkkkitkkkkkkkkkk 


S0S22 

QUEUE 

nYQ2 

TEST  E 

BV$DVA23,1 

SAVEVALUE 

DXL22+,1 

OTX 

MACRO 

LBUS2,BYQ2,XS3EX01 

SAVEVALUE 

EYL2-,1 

UQDQ 

MACRO 

DXQ22,DXL22-,DYQ22,DYL22«,DBP22,X$D£X22,BV$BVA21 

TEST  E 

BVSBVA4,  1 

SAVEVALUE 

KXL2+,1 

UTX 

MACRO 

LBDS2,DYQ22,X$BEX02 

SAVEVALUE 

DYL22-,1 

SPLIT 

1,ACK3 
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COKVSRSiTIONAL  HOSITOR  SISTEB 


TRANSFER  ,STB23 
ACK3  QUEUE  DOQ22 

UTX  BACRO  LBUS2,DOQ22,X$B£XH 

TRANSFER  ,ACK21 

*  STORE- BEHIND  FROM  * 

*  L(2)  TO  1(3)  * 


STB23 

ASSIGN 

11,0 

OQDQ 

BACRO 

KXQ2,KXL2-,KYQ2,KYL2^,KRP2,X$KEX,BV$BVA5 

TEST  E 
SAVEVALOE 

BV$BVA16,1  K3  IN-Q  AND  GBOS  FREE? 

KXL3+,1 

UTX 

BACRO 

GBUS,KYQ2,X$BEXD2 

SAVEVALUS 

KYL2-,1 

UQOQ 

BACRO 

KXQ3,  KXL3-,KYQ3,KYL3  +  ,KRP3,X$KEX,BV$BVA10 

TEST  E 
SAVEVALOE 

BV$BVA17,1  R3  IN-Q  AND  LB0S3  FREE? 

RXL3+,1 

UTX 

BACRO 

LBUS3,KYQ3,X$BEXD2 

SAVEVALOE 

KYL3-,1 

OQD 

BACRO 

EXQ3,RXL3-,,RYL3+,ERP3,X$REX,BV$BVAia 

«««««**«**«* 

*  SERVICED  BY  D31  OR  D32?  • 

44^^^:HL^iim*********s**^‘*******^l!m*** 


TRANSFER  . 5 ,SWS3 1 ,SHS32 

*  SERV.  BY  D33  • 

SHS31  QUEUE  RYQ3 

TEST  E  BV$BVA19,1 

SAVEVALUE  DXL31+,1 

UTX  BACRO  LB0S3,RYQ3,X$BEXD2 

SAVEVALUE  RII,3-,1 
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BQT  HACRO  DXQ3 1  ,I>RP3 1  ,XSDEX31 

SAVEVALDE  DXL31-, 1 

WQT  HACRO  DOQ31,LBOS3,X$BEXH 

TRANSFER  ,ACK22 

*«*«*«««** 

*  SERV.  BY  D32  ♦ 


SVS32 

QOEUE 

RYQ3 

TEST  E 

BV$BVA24,1 

SAVEVALUE 

DXL32+, 1 

OTX 

HACRO 

LG0S3,BYQ3,X$BEXO2 

SAVEVALOB 

RYL3-,1 

OQT 

HACRO 

DXQ32,DBP32,X$D£X32 

SAVEVALOB 

DXL32-,1 

OQT 

HACRO 

D0Q32,LBUS3,X$BEXN 

TRANSFER 

,ACK22 

*  ACK  FEOn  L(2)  TO  L{3)  ♦ 

tiitrlt^m^nmiiint^t******************^!*^ 

ACK22  ASSIGN  11,0 

DQIQ  HACRO  KIQ3, KEP3, KOQ3, X$KEX 

DTX  HACRO  GBUS,KOQ3,X$B£Xn 

OQTQ  HACRO  KIQ2,KRP2,KOQ2,X$K£X 

DTX  HACRO  LBUS2,KOQ2,XSBEXn 

OQTQ  HACRO  B1Q2, RRP2, ROQ2, X$REX 

UTX  HACRO  LBUS2,BOQ2,XSBEXH 

TRANSFER  ,ACK21 

**«*«*« 

*  ACK  FROM  L(2)  TO  L(1)  * 

^^ilt^Hl^t************^^*^****<IHH^iH^1HH^** 

ACK21  ASSIGN  11,0 

OQTQ  HACRO  KIQ2, KRP2, KOQ2,X$KEX 
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VS1J0B  022 

CONVERSATIONAL  MONITOR  SYSTEM 

• 

«TX  MACRO 

GDU5,K0Q2,X$BEXH 

QQTQ  MACRO 

K1Q1,KRP1,K0Q1,X$KEZ 

DTX  MACRO 

LBOS1,KOQ1,X$OEXH 

SPLIT 

1,PN$aiCHA,1 

TERMINATE 

AAA11  ASSIGN 

11,0 

AAA12  ASSIGN 

11,0 

AAA13  ASSIGN 

11,0 

QUEUE 

DIQ11 

SEIZE 

OBP11 

DEPART 

DIQII 

ASSIGN 

4+,X$BEX 

ASSIGN 

7,X$REX 

ADVANCE 

P7 

RELEASE 

DHP11 

TEBHINATE 

«**««***«***»*«««* 

•  AAA12  * 

****4************* 

************* 

*  AAA13  * 

***4i«****«****4>*«4i 


*********************************** 
*  * 

*  TIMER  SEGMENT  -  TIME  UNIT  IS  * 

*  ONE  HANOSECONO  « 

41 4141 4> «  4' *  «  «  *  *  «  *  « li  **  *  4"t>  *  4> «  *  *  «  <1 «  4>  * 

GENERATE  XS7IHES 
TERMINATE  1 


START 

END 
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(star! ) 


(star2) 


(star3) 


(comr) 
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(comr) 
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(star5) 
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(inlZ) 


-28 


(inl3) 
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(stor2) 
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(inl4) 
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(comw) 


(sws21 ) 

r~ 

send  to  D21 
via  lbus2 


D21 

r~ 

send  to  K2 
via  lbus2 


{stb23) 
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(ack21) 


(sws22) 

V 


(stb23) 


{ack32) 


-289- 


(ack43) 


(ack32) 
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FILE:  GPSS54  VS1J0B  Dl 


CONVERSATIOHAL  NONITOB  SYSTEM 


//LAHU  OOB  LAK,MPBOFILE='BETOEM', 

//  PROF  1LF,=*  LOU', 

//  TIME=9 
//♦PASSUOUD 
//GPSS  PEOC 

//C  EXEC  PG«=DAG01,TiaE=6TLIMIT 
//STEPLIB  DD  DSN=PO'rLUCK.LIBEArtt.  GPSS. LOAD, DISP=SHS 
//DOOTPOT  DD  SYS0UT=PB0PILE=RETURN,DCB=BIKSIZE=931 
//DIRTERO  DD  0 KIT=SC BATCH, SPACE* (CYL, ( 1, 1) ) ,DCB=BLKSIZE= 1 880 
//DSYMTAB  DD  U N1T=SCR ATCl{,SPACE=  (CY i,  { 1 ,  1) )  ,DCB=BLKSIZE=71  H 2 
//DREPTGES  DD  U  NIT=SCKATC11,  SPACE*  <CYL ,( 1,1)  ),  0CB=BLKSIZE* 800 
//DIMTUORR  DD  aNIT=SCRATCU, SPACE*  (CYL,  (1,1) ) ,DCB*BLKS1Z£*2680 
//  PEND 

//STEPI  EXEC  GPSS,PARH=C,TLIMIT*9 
//D1NPUT1  DD  * 

REALLOCATE  FU N , 5 , QUE, 1 0, F AC, 50, BVR, 200, BLO , 2000 ,TAB, 50 
'  REALLOCATE  FSV, 50 , HSV , 10,CO»,40000 

•  * 

*  TXN  FARM  USAGE  * 

* 


PI 

CPU  ID  ♦ 

P2 

TXN  ARRIVAL  TIME* 

P3 

TXN  COMPL  TIKE  * 

P4 

TXN  EXEC  TIME  * 

P11 

DUMMY  * 

* 

************  ******** 


************************* 


*  * 

*  MODEL  COMPONENTS  ♦ 

•  * 

*  BOSES:  GBUS,  LBUSI,..  * 

*  CACHES:  D11,...D15  * 

■*  LEVEL  CONTRL;  K1,...K4* 

♦  REQ  PROCS;  B2,  ..  B4  * 

*  DEVICES;  D21,  . . .D42  * 

♦  STORAGE  ;  RI,  RO  • 

•  STORAGE  ;  SI,  SO  * 

♦  STORAGE  ;  TI,  TO  ♦ 

•  STORAGE  ;  AI,  AO  ♦ 

♦  STORAGE  ;  01,  00  ♦ 

*  * 
************************* 

************************* 
♦  ,  * 

*  MODEL  PARAMETERS  * 

*  * 


INITIAL 

INITIAL 


X$MAXHP,10 

X$NEEAD,50C 


DEGREE  OF  HOLTIPROG  PER  CPU 
X  BEAD  REQ 


pile:  gpsssx  vs  1  job  convers&tion&i.  houxtob  systeh 


INITIAL 

X$NWRTT,500 

%  WRITE  KEQ 

INITIAL 

X$PIH1,900 

CONDITIONAL  PROB  OF  FINDING  DATA 

INITIAL 

X$P1N2,900 

IN  A  LEVEL  GIVEN  THAT  THE 

INITIAL 

XJPIN3,900 

DATA  IS  NOT  FOUND  IN  ANY  UPPER 

INITIAL 

X$P1N4,1000 

LEVEL 

INITIAL 

X$POV1,500 

PROB  OF  OVERFLOW 

INITIAL 

X$POV2, 500 

INITIAL 

X$POV3,5C0 

INITIAL 

X5DEX1, 10 

DEVICE  SERVICE  TIME 

INITIAL 

X$DEX2, 100 

INITIAL 

X$OEX3,200 

INITIAL 

XSDEX4, 1000 

INITIAL 

X$BEXM,10  • 

BUS  SERVICE  TIHE 

INITIAL 

XSBEXI, 10 

INITIAL 

XSBEX2,80 

INITIAL 

XSBEX3,320 

INITIAL 

X3REX,20 

DIRECTORY  LOOK  OP 

INITIAL 

X$K£X,10 

CONTROLLER  SERV  TINE 

INITIAL 

X$RDEX1,30 

LOOKUP  PLUS  READ  TIME  OP  CACHE 

INITIAL 

XSTIHEft, 200000 

SIMULATION  TIME 

«*!***«««*«* 

•  4> 

•  SIVEVALUES  * 

«  * 

♦  HTXK  TOTAL  TXM  PBOC.  * 

*  SUMX  TOTAL  EXEC  TIMES  * 

*  sun H  TOTAL  WAIT  TIBES  * 

•  SDHT  TOTAL  ELAPSED  TIH* 

*  * 
***4141  ««•«*««**«««««*««««« 


41  *  *  4i  4' *  41 «  *  *  «  *  «  «  4> «  4^  41 «  «  4i  4>  41 4r « 
*  * 

•  VARIABLES  * 

*  41 
44i  41 41 41  *4>  4i  *  41 4>  4<  4<  41 4>  *  4>  4<  4>  41 4>  4>  4t  41 4> 


HRESP  FVARIABLE 
TXNT  VARIABLE 
TXMW  VARIABLE 
TXKX  VARIABLE 


(XSSUHT/XSNTXH) 

P3-P2 

P3-P2-P4 

P4 


HEAR  RESP  TIRE 
l^XN  ELAPSED  TIME 
TXN  BAIT  TIME 
TXN  EXEC  TIME 


«4i*4i4i  4i**4<*4>4>«4<4i4i4i4<i^4i*4<4>4>4i 
4>  * 

♦  TABLES  * 

♦  41 

«  «  41 41 4>  4' 4i  41 41  *  4< «  4>  *  4<  41 41 4<  4> «  41 4<  4^  4>  * 


TXNT  TABLE 
TXKW  TABLE 
TZHX  TABLE 


VSTXNT, 100, 100, 100 
V$TXNW, 100,100,100 
V$TXNX,100,100, 100 


4i*4i4>*4i4i4i4t4>*4i4i*4>4i4>«4i4i4t«4i4i4i 
*  * 
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CONVERSATIOHAL  MONITOB  SYSTEM 


♦  FUNCTIONS  ♦ 

*  * 

WICHW  FUNCTION  P1,D5 
2,WWH11/3,WhW12/4 ,WHW13/5,WHW14/6,WHW15 


WICHA  FUNCTION  P1,D5 
2,AAA11/3, AAA  12/4, AAA  13/5, AAAI  4/6, AAAI  5 


t^ni^itm******************* 


*  * 

*  STORAGE  FOE  L(1)  • 

♦CACHES  ♦ 

♦  ♦ 


««  4i  4t  ^  4i  « 


STORAGE  S$H ID11 , 1 0/S$SI Dll, 2/S$TIDl 1 , 10/S$AID1 1,10 
STORAGE  S$RID12,10/S$SID12,2/S$TID12, 10/S$AID12, 10 
STORAGE  S$RID13, 10/S$SID13,2/S$TID13, 10/SSAI D1 3, 1 0 
STORAGE  S$EID14, 10/S$S1D14,2/S$TID14, 10/S$AID1 4, 10 
STORAGE  S$BID15,10/S$SID15,2/S$TID15, 10/S$AID15, 10 


******  ************** 
♦  ♦ 

•  STORAGE  FOR  DEVICES  ♦ 
«  ♦ 

************************* 


/ 


STORAGE  SiRID21,10/S$SID21, 10/ ^$TID21,10 

STORAGE  S$RID22, 10/S$SIP22, 10/ J$TID22,  10 

STORAGE  S$EID31, 10/S3SIDJ1, 10/S$TID3 1 ,1  0 

STORAGE  S$RID32, 10/S$SID32, 10/S$TID32, 10 

STORAGE  SJ RID41, 10/S3SID41, 10/S$TID41,10 

STORAGE  S$RID42, 10/SJSID42, 10/S$TID4 2, 10 

************************* 


*  * 

♦  STORAGE  FOR  EEQ  PROC  ♦ 

♦  ♦ 
************************* 


STORAGE  S$R IR2 , 1 0/S$SIR  2, 10/S$TIR2, 1  0/S  $A1R  2, 1 0/SSOIR2, 1 0 
STORAGE  S$RIE3, 10/S$SIR3, 10/S$Tin3, 10/S$AIR3, 10/S$OIR3,10 
STORAGE  S$EIB4, 10/S$SIE4, 10/S$TIE4, 10/S$AIR4, 10/S$OIR4,10 


************************* 
*  * 

♦  STORAGE  FOR  K1  * 

*  * 
************************* 


STORAGE  S$ROKl , 1 0/S$SOK 1, 10/S$TIK 1, 1 0/S$AIX 1 , 1 0/S$OOK1 , 1 0 

************************* 

*  * 
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STORAGE  FOE  K2, K3,K4  ♦ 

* 


STORAGE 

STORAGE 

STORAGE 

STORAGE 

STORAGE 

STORAGE 


S$RIK2,10/S$SIK2, 10/S$TIK2, 10/SSAIK2,10/S$OIK2,10 
SJRIK3,10/S$SIK3,10/S$T1K3,10/S$AIK3, 10/S$OIK3,10 
Si;RIK4,  10/r>$SIK4,  10/.*>$TIK4,  10/S$AIK4,  10/S$OIK4, 10 
S$EOK2, 10/S$SOK2, 10/S$TOK2, 1 0/S$AOK2 , 10/S$OOK2, 10 
S$ECK3, 10/S$SOK3, 10/S5TOK3,  10/S$AOK3, 10/S$OOK3, 10 
S3:E0K4,  10/S$SOK4, 10/S$TOK4,1  0/S  $AOK4 ,1  0/S$OOK4 , 10 


t*«***iiiii***ttiniii*t  ******* 
*  * 

*  BOOLEAN  VARIABLES  * 

*  * 
*****  ******************** 


************************* 
*  * 

*  BV  FOR  READ-THROUGH  ♦ 

*  * 
************************* 


ETOK2  BVAkIABLE 
BTOK3  BVARIABLS 
RT0K4  BVARIABLE 


FND$GB0S*SNF$TIK1 

FNUSGBUS*SNF$TIK1*SNF$TIK2 

FNU$GBUS*SNF$TIK1*SNF$TIK2*SNF$TIK3 


************************* 
*  * 

*  BV  FOR  L{1)  ♦ 

♦  ♦ 
************************* 


DKE1  BVARIABLE 
DKSl  BVARIABLE 
DK01  BVARIABLE 
KDT11  BVARIABLE 
KDT12  BVARIABLE 
KDT13  BVARIABLE 
KDT14  BVARIABLE 
KDT15  BVARIABLE 
KDA11  BVARIABLE 
KDA12  BVARIABLE 
KDM3  BVARIABLE 
KDA14  BVARIABLE 
KDA15  BVARIABLE 


FNU$LBU31*SNF$R0K1 

FNUSLDOS 1*SNF$SOK1 

FNU$LDUS1*SNF$OOK1 

FNU$LB031+SNFITID11 

FNU$LBUS1*SNF$TID12 

FNUSLDUS1*SNF$TID13 

FNU$LB0S1*SNF$TID14 

FNU$LB0S1*SNF$TID15 

FN0$LaU31’t'SNF$AIDl1 

FNUSLBUS1*SNF$AID12 

FifU$LBUS1*SNP$AID13 

FNUSLBUS1*SNF$AID14 

FNU$LDUS1*SNF$AID15 


************************* 

*  * 

*  BV  FOR  INTER  LEVEL  COK* 

*  ♦ 
************************* 

KKR12  BVARIABLE  FNU$GBaS+SNP$RIK2 
KKS12  BVARIABLE  FNOSGBU 3«SN F$SIK2 
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KK012  BVABIABLE 
KKT21  BVABIABLE 
KKA21  BVAFIABLE 
KKB23  BVABIABLE 
KKS23  BVABIABLE 
KK023  BVABIABLE 
KKT32  BVABIABLE 
RKA32  BVABIABLE 
KKIt34  BVABIABLE 
KKS34  BVABIABLE 
RK034  BVABIABLE 
KKT43  BVABIABLE 
KKA43  BVABIABLE 


FNU$GBUS*SNF$SOK2 
P(I0$GBa3*SNP$TIK1 
FNt}3:GUUS*SMFS  AIK1 
FNU$GBUS*SWF$RIK3 
FNU$GDtIS*SNFJSIK3 
FNO$GDOS*SNF$OIK3 
FNU$GBUS*SHF$TIK2 
FNUBGD0S*SNF$AIK2 
FNU$GBaS*SNF$BIR4 
FHU3GBUS«SNF$SIK4 
FNU$dBUS«SNF$OIK4 
FNUSGDUS* SNF$TIK3 
FNa$GBUS*SMF$AIK3 


*•<1**  ******************** 
*  * 

♦  BV  FOB  L(2)  OPS  * 

♦  * 
4i4i«4i1i  «*«#««**«****«****** 


KBB2  BVABIABLE 
KRS2  BVABIABLE 
KBT2  BVABIABLE 
KRA2  BVABIABLE 
KR02  BVABIABLE 
BDR21  BVABIABLE 
EDS21  BVABIABLE 
BDT21  BVABIABLE 
BDB22  BVABIABLE 
BDS22  BVABIABLE 
BDT22  BVABIABLE 
DKS2  BVABIABLE 
DKT2  BVABIABLE 
DKA2  BVABIABLE 
BKR2  BVABIABLE 
FKP2  BVABIABLE 
BKA2  BVABIABLE 


FNU$LBOS2*SNr$EIB2 

FKU£LDUS2*SNFSSIK2 

FNU$LB0S2*SNF$T1E2 

FNUSLDUS2*SHF$AIE2 

FNU$LDUS2*SNF$0IE2 

PNU$1BUS2*SNF$EID21 

FNU$LB0S2*SNF$SID21 

FNU$LBUS2*SNF$TID21 

FNU$LDUS2*SNF$RIB22 

FNUJLB0S2*SKF$SIE22 

?[IUJLBOS2*SHF$TID22 

FNU$LBUS2*SNF$S0K2 

FSn$LDUS2*SNP$TOK2 

FNU$LBOS2*SSFJAOK2 

FNU$LBUS2*SNF$ROK2 

FllU$LBUS2*SNF$OOK2 

FNUSLBUS2«SNF$AOK2 


nf^Hfrn*************  ******** 
*  * 

*  BV  FOR  L(3)  OPS  * 

*  * 
************************* 


KRB3  BVABIABLE 
KRS3  BVABIABLE 
KBT3  BVABIABLE 
KRA3  BVABIABLE 
KB 03  BVABIABLE 
BDR31  BVABIABLE 
BDS31  -BVABIABLE 
BDT31  BVABIABLE 
BDB32  BVABIABLE 
BBS32  BVABIABLE 
BDX32  BVABIABLE 


Ftia$LBBS3*SNF$BIR3 

FM0$LB0S3*SNF$SIB3 

FN0$LB0S3*SNF$TIB3 

FN0$LB0S3*SHF$AIR3 

FNU$LBUS3*SIIF$0IR3 

FB0$LBUS3*SNF$RID31 

FNU$LBUS3*SNF$S1D31 

FNUXLBUS3*SIIFJTID31 

FI10SLBUS3*SNF$RID32 

FN0$LBUS3*SNF$SID32 

FHD$IBUS3*SNFSTI032 
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DKS3  nVARIABLE 
DKT3  UVARIABLE 
DKA3  BVAEIABLE 
PKB3  DVARIABLE 
RKA3  BVAEIABLE 
EK03  BVAEIABLE 


FNnJLniJS3>SNF$SOK3 

KNUJLIJ033*SNF$TOK3 

FNU$LBOS3*SHF$AOK3 

FNU$LBU33*SNF$E0K3 

FNUJiLBUS3*SNF$AOK3 

FNUSLBOS3*SNF$OOK3 


41 41  « :ti «  #  4i  4c  «  «  *  *  «  ««  *  «  * 

•  * 

*  BV  FOE  L(4)  OPS  * 

*  <1#  «  «  4141  *4c  41 4c  4c «  4<  4c  4>4i  *  etc  4c «  *  4< «  4c 


KRR4  BVAEIABLE 
KES4  BVAEIABLE 
KB04  BVAEIABLE 
5BR41  BVAEIABLE 
EDS41  BVAEIABLE 
BDR42  BVAEIABLE 
RDS42  BVAEIABLE 
DKT4  BVAEIABLE 
DKA4  BVAPIABLE 


FHU$LBUS4*SNP$EIE4 

rN0$LB0S44‘SNF$SlR4 

FNU$LBUS4*5NF$OIE4 

FNUSLBUS4*SNF$EID41 

FNU‘EL3US44cSNF$SID41 

FNU5LBUS44'SNF$RID42 

FN0$LBUS44SNF$SID42 

FNU$LBUS44>SNF$TOK4 

FNU$LBUS4*SNF$AOK4 


41  #  4c  41 4c  4c  4c  4c  4' 4>  4c  4c  4c  4c  4' 4c  4c  4>  41 4>  4c  4c  4c  4> 

♦  41 

41  MACROS  ♦ 

41  4c 

c4  4c  4c  41  **  4>  4c  *  4c «  4c  4c  41 4  4c  4c  4  4  4c  4c  41 4c  4c  4c 


4414c 44  4444  444*4c41c44f44*4444 


4  * 

*  MACRO  -USE  * 

♦  tA  FACILITY  * 

•  *B  USAGE  TIME  * 

4  41 


444  4444  *4  *444  4  *41 44*4  44  444 


USE  STARTHACRO 

SEIZE  *K 

ADVANCE  *B 

ASSIGN  4+,#B 

RELEASE  dA 

ENDM ACBO 


4444444444444444444444444 


4  * 

*  MACRO  -  SEND  * 

4  * 

4  tA  FROM  * 

4  #B  TO  ♦ 

4  #C  VIA  ♦ 

*  ID  TRANSIT  TIME  * 

*  #E  BV  FOE  SEND  OP  * 

4  * 


4444444444444444444444444 
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SEED 


STARTMACRO 

TEST  E 

»E,1 

ENTER 

«B 

SEIZE 

«C 

ADVANCE 

#D 

ASSIGN 

4^, ID 

RELEASE 

#C 

LEAVE 

*A 

ENDMACRO 

***************** 

*  * 

•  EACEO  -  PINI  • 

*  .  '  * 
*i|i«*«******««ii4t«4i<tt**«**4i« 


FZEI  STABTHACRO 
HARK 

SAVEVALUE 

SAVEVALOE 

SAVEVALOB 

SAVEVALUE 

SAVEVALUE 

ASSIGN 

ASSIGN 

ASSIGN 

ASSIGN 

ENDHACRO 


3 

NTKN*,1 
SUHX«,V$TXNX 
SU  NH-^.VSTXNW 
SUaT+,VJTXNT 
HRESPfVSHBESP 
1,0 
2,0 
3,0 
4,0 


BEGIN  SIMULATION 


SIHULATB 

*  * 

•  CPU  #1  ♦ 

*  * 
^t^tti********>lf************ 


RBULT  3,5,7,9,11,13,15,17 


CPUI  GENERATE 
STAH1  PRIORITY 
MARK 
ASSIGN 
TRANSFER 
BBR1  TRANSFER 


,,,X$nAXHP,,,F 

9  SET  HIGH  P  FOR  NEH  TXH 

2  ARRIVAL  TIME 

1,1  CPU  ID 

.X$NP.£AD,NHN1,BRR1 
.X$PIN1,NIN11,EIN11 


♦  * 
«  DATA  IS  IN  DATA  CACHE  * 
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*  « 

tf^^nnnmni^t^^niM*********^** 


P.I811  ENTER 
USE  MACRO 
LEAVE 
FlHI  MACHO 

TRANSFER 


RID1 1 

DRP11,X$RDEX1 
BID1 1 

, START 


POT  TXN  IN  READ  ESQ  BOFFEB 
SEARCH  AND  READ  CACHE 
FREE  BUFFER 

A  NEH  TXN 


•  * 

•  DATA  IS  NOT  IN  CACHE  • 

*  I* 
^^0tt*********i****if******* 


HIHT1  ENTER 
-USE.  MACRO 

PRIORITY 
SEND  MACRO 


EID11 

DRP11,X$REX 

0 


POT  IN  BEAD  REQ  BUFFER 
SEARCH  DIRECTORY 
BESET  PRIORITY 


EID11,nOK1,LBOS1,X$BEXH,DV$DKR1 


TRANSFER  ,COtlR 


TO  COMMON  CODE  FOR  BEAD 


41 4t « **««««*«  *«*  O  4> « 

«  * 

*  WRITE  REQUEST  TO  CACHE* 

*  « 


8HH1 

ENTER 

SID11 

PUT  ^  XN  IN  WRITE  REQ  BUFFER 

USE 

MACRO 

ORPn,X$RDEX1 

HBI:E  data  in  CACHE 

SEND 

PRIORITY 

MACRO 

0 

SID11,SOK1,LBOS1 

RESET  TXN  PRIORITY 
,X$B£X1, BVSDKSI 

SPLIT 

1,COMH 

FINI 

MACRO 

TRANSFER 

,STAB1 

A  NEH  TXN 

#41*4141  ******************** 
*  * 

*  CPU  »2  * 

*  * 
************************* 


CP02  GENERATE 
STAB2  PRIORITY 
MARK 
ASSIGN 
TRANSFER 
RBR2  TRANSFER 


,,,XtMAXMP,,,F 

9  SET  HIGH  P  FOR  NEH  TXN 

2  ARRIVAL  TIME 

1,2  CPU  ID 

.XSNREAD,HHW2,RBR2 
.X$?IN1,N1N12,BIN12 


************************* 


-299 


rxlS:  GPSS54  VS1J0B  Dt 


COHVEBSATIOMAL  HOHXTOR  STSTSH 


«  * 

*  DATA  IS  IN  DATA  CACHE  ♦ 

4>  « 

*************** 


RIN12  ENTER  RID12 

DSE  MACRO  DRP12,X$BDEX1 

LEAVE  R1D12 

FINI  MACRO 

TRANSFER  ,STAR2 

************************* 

*  * 

*  DATA  IS  NOT  IN  CACHE  * 

*  * 
************************* 


PUT  TXM  IN  READ  EEQ  BUFFER 
SEARCH  AND  BEAD  CACHE 
FREE  BUFFER 

A  NEH  TXN 


HIN12  ENTER 
DSE  MACRO 

PRIORITY 
SEND  MACRO 


RID12 

DRP12,X$REX 

0 


POT  IN  BEAD  BEQ  BUFFER 
SEARCH  DIRECTORY 
RESET  PRIORITY 


EID12,R0K1,LBUS1,X$BEXM,BV$DKH1 


TRANSFER  ,COKR 

*  * 

•  WRITE  REQUEST  TO  CACHE* 

*  * 
************************* 


TO  COMMON  CODE  FOR  BEAD 


HUK2 

ENTER 

SID12 

PUT  TXN  IN  WRITE  REQ  BDFFEB 

DSE 

MACRO 

DEP12,X$RDEX1 

WRITE  DATA  IN  CACHE 

SEND 

PRIORITY 

MACRO 

0 

SID12,S0K1,LBUS1 

RESET  TXN  PRIORITY 
,X$DEX1,BV$DKS1 

SPLIT 

I.COMW 

FINI 

MACRO 

TRANSFER 

,STAE2 

A  NEW  TXN 

************************* 
*  * 

*  CPU  #3  ♦ 

*  * 
************************* 


CP03  GENERATE 
STAR3  PRIORITY 
MARK 
ASSIGN 
TRANSFER 
BBB3  TRANSFER 


,, ,X$aAXMP,,,F 

9  SET  HIGH  P  FOR  NEW  TXN 

2  ARRIVAL  TIME 

1,3  CPU  ID 

.X$NHEAD,WHW3,RHR3 
.X$PIN1,NIN13,EIN13 
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♦  * 

*  DATA  IS  IN  DATA  CACHE  * 

*  * 
t********* *************** 

BIN  13  ENTER  RID13 

USE  HACBO  ORP13,X$BDEX1 

LEAVE  BID13 

FIN I  HACBO 

TRANSFER  ,STAR3 

************************* 

*  * 

*  DATA  IS  NOT  IN  CACHE  * 

*  * 

************************* 

NIN13  ENTER  BID13  POT  IN  BEAD  BEQ  BDFFER 

OSB  HACBO  DRP13,X$R£X  SEARCH  DIEECTOBY 

PRIORITY  C  BESET  PBIOBITY 

SEND  HACBO  BIDl 3, BOKI ^LBUSl , X$B£XH,BV$DKR1 

TRANSFER  ,COnH  TO  COHHON  CODE  FOB  BEAD 

************************* 

*  * 

*  NBITE  BEQUEST  TO  CACHE* 

*  .  * 

N8H3  ENTER  SID13  POT  TXN  IN  HBITE  BEQ  BUFFER 

USE  HACBO  DBF13,X$R0£X1  NBITE  DATA  IN  CACHE 

PRIORITY  0  RESET  TXN  PRIORITY 

SEND  HACBO  SID13,S0K1,LBUS1,X$B£X1,BV$DXS1 

SPLIT  1,COHN 

FINI  HACBO 

TBANSFEB  ,STAB3 

************************* 

*  * 

*  CPU  #4  * 

♦  * 

************************* 

CP04  GENERATE  , , .XSHAXHP, , , F 

STAB4  PRIORITY  9  SET  HIGH  P  FOB  NEB  TXN 

HABK  2  ARRIVAL  TIHE 


PUT  TXN  IN  READ  BEQ  BUFFER 
SEARCH  AND  READ  CACHE 
FREE  BDFFER 

A  NEW  TXN 
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ASS1G8  1,4  CPO  ID 

TBANSFEB  . XSNBEAD, UVH4 ,BBB4 

RRB4  TBANSFEB  .X$PXH1, 1IIB14,BXN14 

****«*«*i|i«*««*iti*««4i4i*«*** 

*  * 

*  DATA  XS  IN  DATA  CACHE  * 

*  ■  '  * 

BIN14  ENTER  BID14 

OSE  HACBO  DBF14,X$BOEZ1 

LEAVE  BX014 

FXNX  HACBO 

TBANSFEB  ,STAB4 

*******  «*«*****««««* 

•  « 

*  DATA  XS  NOT  IN  CACHE  * 

*  * 
t******************* 

NXN14  ENTER  BXD14  POT  IN  BEAD  BEQ  BOFFEB 

OSE  HACBO  DRP14,X$BEX  SEABCU  DXBECTOBY 

PRIORITY  0  BESET  PBIOBITY 

SEND  HACBO  BIDl 4,E0K1 ,LB0S1 , X$B£XH,BV$DKB1 

TRANSFER  ,COHE  TO  COHMOK  CODE  FOR  BEAD 

lf^t****************Mf***** 

*  * 

*  HBITE  BEQUEST  TO  CACHE* 

♦  * 

«**********««**«****»**<l>* 

HNN4  ENTEB  SXD14  PUT  TXN  IN  HBITE  BEQ  BOFFEB 

USE  HACBO  DBP14,XSR0EX1  HBITE  DATA  IN  CACHE 

PBIOBITY  0  BESET  TXN  PBIOBITY 

SEND  HACBO  SXD14,SOK1 ,LBOS1 ,X$BEX1 , BVSDKSI 

SPLIT  1,COHN 

FINI  HACBO 

TRANSFER  ,STAB4  A  NEH  TXN 

tt*********************** 

♦  * 

*  CPO  #5  ♦ 

*  * 
ti***********************’* 


PUT  TXN  IN  BEAD  BEQ  BUFFER 
SEABCH  AND  BEAD  CACHE 
FREE  BUFFER 

A  HER  TXN 
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CPUS  GENERATE  , , , X$«AXHP, , ,F 

STABS  PEIOBITX  9  SET  HIGH  P  FOB  NE3  TXN 

BARK  2  ARRIVAL  TIME 

ASSIGN  1,5  CPO  ID 

TRANSFER  . X$N R EAD, WHW5, EEB5 

ERRS  TRANSFER  .XSPIN 1 , HIN 15, BIN  15 

*  * 

*  DATA  IS  IN  DATA  CACHE  ♦ 

*  * 

*t.^^*^^****^***1^*^^**^^*^^**i^^* 


EIN15  ENTER  PID15 

USE  MACRO  DEP15,X$RDEX1 

LEAVE  RID15 

FIHI  MACRO 

TRANSFER  , STARS 


POT  TXN  IN  READ  REQ  BDFFER 
SEARCH  AND  BEAD  CACHE 
FREE  BUFFER 

A  NEW  TXN 


*  * 

♦  DATA  IS  NOT  IN  CACHE  * 

*  * 


NIN15  ENTER  RID15  PUT  IN  READ  REQ  BOFFER 

OSE  MACRO  DBP15,X$REX  SEARCH  DIRECTORY 

PRIORITY  0  RESET  PRIORITY 

SEND  MACRO  RID1 5, BOK 1, LB  OS  1 , XSBEXM, BVSDKBI 

TRANSFER  ,COME  TO  COMMON  CODE  FOB  READ 


*«********:**««««  4>  * 

*  * 

*  WRITE  REQUEST  TO  CACHE* 

*  ♦ 

******************** 


BWH5  ENTER  SID15  POT  TXN  IN  WRITE  REQ  BUFFER 

OSE  MACRO  DRP15,X$BDEX1  WRITE  DATA  IN  CACHE 

PRIOBITY  0  BESET  TXN  PRIORITY 

SEND  MACRO  SID  1 5, SOK 1,LBOS 1 , XSBEX 1, B V$DKS1 

SPLIT  1,COHW 

FIHI  MACRO 

TRANSFER  , STARS  A  NEW  TXN 


*  * 

*  COMMON  CODE  FOR  READ  BEQUEST  * 
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COHR  ASSIGN  11,0  . 

OSE  EACRO  KRP1,X$KEX 

SEND  MACRO  ROKI ,RlK2,GB0S,XiBEXH, BVSKKR 12 

OSE  MACRO  KRP2,X$K£X 

SEND  MACRO  RIK2 , RIB2 ,LBUS2, XSBEXH, BV$KBB2 

OSE  MACRO  £RP2,X$REX 

TRANSFER  .X$P1N2, NIN2, EIN2 
NIN2  -ASSIGN  11,0 

SEND  MACRO  RIR2,BOK2,LBUS2, XSBEXM, BV$RKR2 

OSE  MACRO  KRP2,XSKEX 

SEND  MACRO  ROK2,BIK3,GBas, X$BEXM,BV$KKR23 

OSE  MACRO  KRP3,X$KEX 

SEND  MACRO  BIK3 , RI R3, LBOSJ , XSBEXH , BV$KRR3 

OSE  MACRO  RRP3,X$BEX 

TRANSFER  .X$PIN3,NIH3,RIN3 
NIN3  ASSIGN  11,0 

SEND  MACRO  BIR3,R0K3,LB0S3,XS0EXM,BV$BKB3 

OSE  MACRO  KRP3,X$KEX 

SEND  MACRO  BOK3,RIK4 , GBOS, X$BEXM,BV$KKE 34 

OSE  MACRO  KRP4,X$KEX 

SEND  MACRO  BIK4 , RI R4, LBO S4,XSBEXH, BV$KRR4 

OSE  MACRO  RRP4,X$REX 

TRANSFER  ,RIN4 


♦ - - - - « 

*  * 

*  READ  DATA  IS  FOOND  IN  L(2)  ♦ 

*  * 

- - - - - — - - — - — 


E1H2  TRANSFER  . 5,RBR21 ,BRR22 
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*  «  41 1 4>  *  4' ««<*«  « 

*  ♦ 

*  DATA  IS  IN  D21  * 

*  * 

BBR21  ASSIGN  11,0 

SEND  HACBO  BI R2 ,BI D2 1 ,LBUS2, XSEEXH, BVSBDB2 1 

OSE  NACBO  0BP21,XS0EX2 

SEND  BACBO  BXD21,TOK2^LBOS2,X$BEX1,BV$DKX2 

TBAHSFER  ,ETF2 

^tf*********  ************** 

*  * 

*  DATA  IS  IN  D22  * 

*  * 

************************* 

BBE22  ASSIGN  11,0 

SEND  HACBO  BIB2, BI D22, LBUS2, X$0EXH, BV$BOB22 

OSE  BACBO  DBP22,X$D£X2  ' 

SEND  HACBO  BI022,T0K2,LBUS2, XSBEXI ,BV$DKT2 

TRANSFER  ,HTF2 

♦  ♦ 

♦  READ-THROOGH  TO  L(1)  ♦ 

*  * 

♦♦♦***♦*»*♦♦*♦**♦♦♦* 

BTF2  ASSIGN  11,0 

OSE  HACBO  KBP2,X$REX 

SEND  HACBO  TOK2, TIK 1 ,GBOS, X$BEX 1 , BV$BT0K2 


* 

•  STORE  DATA  INTO  L{1)  AS  EESDLT  OF  A  BEAD-THROUGH 

* 


STOE1  ASSIGN  11,0 

USE  HACBO  KBP1,X$KEX 

SPLIT  1,FN$WICUN,1 
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TERMINATE 

*  4> «  *  *  *  4«<<  4c «  *  «  «  *  « >tr  4t «  4ci|r  4: 

«  41 

*  RT  STORE  INTO  Dll  • 

*  * 
♦  *♦♦♦**♦  4c  ♦♦  4c*  **♦♦*♦  + 4t  *  iji  41# 


SWH11  ASSIGN 

SEND  MACRO 

USE  MACRO 

TRANSFER 
NOVll  LEAVE 

FINI  MACRO 

TRANSFER 

OVL11  SPLIT 

PIHI  MACRO 

TRANSFER 
OVF11  ASSIGN 

SEND  MACRO 

TRANSFER 


11,0 

TIK1,TID11,LBUS1,X$BEX1, BVSKDTl 1 

DRP11,X$DEX1 

.X$P0V1,N0V11,0VL11 

TID11 

,STAR1 

I. OVFII 

,STAR1 

II, 0 

TID11,00K1,LB0S1,X$BEXM,BV$DK01 

,0VL1 


****  4c  4c  *  4c  «*  4c  4c  41  *  4c  4c «  4c  *  41 4c  4c  4c  4c  41 
4c  4c 

•  RT  STORE  INTO  D12  4- 

4c  4c 

41 4<  4c  4c  4<  *  4c  4c  4c  4>  4c  *  4>  4c  4>  41 41 4' 4c  4<  4c  4>  4>  4>  * 


8HW12 

ASSIGN 

11,0 

SEND 

MACRO 

TIKI, TIDI 2, LBUS 1, XSBEXI, BVSKDTl 2 

OSE 

MACRO 

DRP12,XSDEX1 

MOV  12 

TRANSFER 

LEAVE 

.X$POV1,NOV12,OVL12 

TID12 

PIKI 

MACRO 

TRANSFER 

,STAB2 

OVL12 

SPLIT 

1,OVF12 

FINI 

MACRO 
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Difc 


TRANSFER  ,STAR2 

OVF12  ASSIGN  11,0 

SEND  MACRO  TID1 2, OOK1,LBOS1 ,X$DEX«, B V$DK01 

TRANSFER  ,0¥L1 

t************************ 

*  • 

*  RT  STORE  INTO  D13  ♦ 

*  ♦ 

BHHI 3  ASSIGN  11,0 

SEND  MACRO  TI Kl , TI D13, LBOS 1 , X$BEX 1, B VSKDTl 3 

OSE  MACRO  DRP13,X$DEX1 

TRANSFER  . XSPOVI ,NOV13,OVL13 
H0V13  LEAVE  TID13 

PINI  MACRO 

TRANSFER  ,STAE3 

OVL13  SPLIT  l,OVF13 

FINI  MACRO 

TRANSFER  ,STAH3 

OVP13  ASSIGN  11,0 

SEND  MACRO  TID1 3 ,OOK 1 , LBOS 1 , X$BEXM, BVSDKOI 

.TRANSFER  ,0VL1 

♦  ♦ 

*  RT  STORE  INTO  D14  ♦ 

*  * 

« «« 4i t  *  4>iti *«««** 41 « 

HWH14  ASSIGN  11,0 

SEND  MACRO  TIKI , TI D1 4, LBOS 1 , XSBEXI , BVSKDTI 4 

OSE  MACRO  DRP14,X$DEX1 

TRANSFER  . X SPOV 1 , NOV  14, OVL 14 

NOV14  LEAVE  TID14 

FINI  MACRO 

TRANSFER  ,STAR4 
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OVL14  SPLIT  1,OVF14 

FINI  MACRO 

TRANSFER  ,STAE4 
OVF14  ASSIGN  11,0 

SEND  MACRO  TI D  1 4, OOK 1 , LBUS 1 , X$ BEXM, BV$ DK01 

TRANSFER  ,OVL1 

*  * 

*  ET  STORE  INTO  D15  * 

*  ♦ 

^ltiti*t:*:******>{f***t****'lf*** 

SMH15  ASSIGN  11,0 

SEND  MACRO  TIKI , TI D 1 5, LBUS 1 , X$BEX 1, BV$K DTI  5 

USE  MACRO  DRP15,X$DEX1 

TRANSFER  . X$ PO V 1 , NOV  1 5,0 VL 15 
NOV15  LEAVE  TID15 

FINI  MACRO 

TRANSFER  , STARS 

OVL15  SPLIT  1,OVF15 

FINI  MACRO 

TRANSFER  , STARS 

OVF15  ASSIGN  11,0 

SEND  MACRO  TID1 5,COK 1 , LBUS 1 , XSBEXM, BV$D K01 

TRANSFER  ,OVL1 

♦  * 

*  HANDLE  OVF  FROM  L ( 1)  ♦ 

*  ♦ 

OVL1  ASSIGN  11,0 

USE  MACRO  KRP1,X$KEX 

SEND  MACRO  OOK 1 ,OIK2 ,GBU3, XS BEXM , BVSKKO 1 2 

USE  MACRO  KBP2,X$KEX 
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SEND  HACBO  01K2,OIB2,LBO  S2,  X$BEXn«BV$KIt02 

USE  HACBO  .  BRP2,X$REX 

LEAVE  OIB2 

TERHINATE 


* 

*-  BEAD  DATA  IS  FOUND  IN  L(3) 

4> 


BIH3  TBANSFEB  . 5,BBfi31,BRB32 

«  * 

♦  DATA  IS  IN  D31  ♦ 

«  * 

*4t*4>4>*4it**4i***«’<i*^:»**4i**« 

BBB31  ASSIGN  11,0 

SEND  MACRO  RIB3,RID31,LBDS3,XSBEXH,BVSRDR3 1 

USE  HACBO  DRP31,X$0EX3 

SEND  MACRO  RID31, T0K3,LBUS3, XSBEX2,BV$DKT3 

TRANSFER  ,ETF3 

0*^*4******************** 

♦  * 

*  DATA  IS  IN  D32  * 

*  * 

4444444*44*4***4**4*4***4 

BRB32  ASSIGN  11,0 

SEND  MACRO  BIB3 ,RID32, LBUS3, XSBEXH, BV$BDB32 

USE  MACRO  DRP32,X$DEX3 

SEND  MACRO  RI932,T0K3,LB0S3,X$BEX2,BT$DKT3 

TRANSFER  ,RT?3 

*  * 

♦  BT  TO  L{1)  AND  L(2)  ♦ 

♦  *■ 
4444*4******************* 

BTP3  ASSIGN  11,0 


-309 


FILE:  GPSSS4  VS1J0B  D|^  COKVERS&TIOSAL  BOMXIOB  SYStBB 


MACRO 

KRP3,X$KEX 

TEST  E 

BV$ETOK3,1 

ENTER 

TIKI 

ENTER 

TIK2 

SEIZE 

GBUS 

ADVANCE 

X$BEX2 

ASSIGN 

4«,X$B£X2 

RELEASE 

GBUS 

LEAVE 

TOK3 

SPLIT 

1,STOR1 

SPLIT 

1,ST0H2 

TERMINATE 

\ 

STORE  DATA  INTO  L(2)  AS  RESULT  OF  A  READ-THROUGH 


5T0R2  ASSIGN 

USE  MACRO 

SEND  MACRO 

USB  MACRO 

SPLIT 

TRANSFER 


11,0 

KRP2,X$KEX 

TXK2,TIR2,LBaS2,X$BSX2,BVSKBT2 

RBP2,XSREX 

1,OVH2 

.5,SSS21,SSS22 


^^^^m^l^^^t^nt^t****^^***^>*l^**** 
*  * 

*  STORE  INTO  D21  • 

*  * 


SSS21  ASSIGN 

SEND  MACRO 

USE  MACRO 

LEAVE 

TERMINATE 


11,0 

TIR2 , TID2 1, LB0S2,X$BEX2, BVSRDT2 1 

DBP21,XSDEX2 

TI021 


*41******  **«**«  ***«***«**« 
*  ♦ 

•  STORE  INTO  D22  • 

*  * 
************************* 

SSS22  ASSIGN  11,0 
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SEND  MACRO  Tin2,TI022,LDUS2«X$B£X2,DV$RDT22 

OSE  MACRO  DRP22,X$D£X2 

LEAVE  TID22 

TERMINATE 

*  * 

*  OVERFLOW  IIAHOLING  * 

^itt********************** 


0VH2 

0VL2 

TRANSFER 
TEST  E 
ENTER 

SEIZE 

ADVANCE 

ASSIGN 

RELEASE 

.X$P0V2,N0VL2,0VL2 

BVSEK02, 1 

00K2 

LBUS2 

XSBSXn 

4+,XSUEXH 

LBUS2 

SEND 

MAC  BO 

00K2 ,0IK3 ,GBOS, XSBEXH , BYSKKO  23 

DSE 

MACRO 

KRP3,X$K£X 

SEND 

MACRO 

0IK3.0IR3,LB0S3,X$BEXM,BV$KB03 

DSE 

MACRO 

BRP3,X$REZ 

M0VL2 

LEAVE 

TERMINATE 

01 B3 

* - -  —  - - - — 

♦ 

*  BEAD  DATA  IS  FOOHD  IN  L (4) 

* 


* 

* 

* 


RIM  4 

TRANSFER 

.5,Bfifi41,RBR42 

******************** 

*  * 

*  DATA 

IS  IN  D41 

* 

• 

♦ 

4i  4t «  4c  «  «  *  «i|i «  4r :(( «  4t  ♦  «  «  ♦  « 

BBR41 

ASSIGN 

11, D 

SEND 

MACRO 

BIR4 , BID4 1, LB0S4,XSBEXH, BVSB0B4 1 

OSE 

MACRO 

DRP41,X$DEX4 

SEND 

MACRO 

RID41,T0K4,LBUS4,X$B£Z3,BV$DKT4 

TRANSFER 


BTF4 
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*  ««*««*  4t  *  41*41  >t> «  ****>»*  *4> « 

*  • 

♦  DATA  IS  IH  D42  * 

•  ♦ 

41  *  *  *  4>  *  4)  *  1 4i  *  4>  4I  *  4>  4>  *  4>  4I  *  4>  *  «  *  * 

BBR42  ASSIGN  II.O 

SEND  MACRO  BIR4,BZ042,I.BaS4,X$BEZH,BV$B0B4  2 

DSB  MACRO  DBP42,X$DEX4 

SEND  MACRO  BID42,TOK4,LBBS4,XSBEX3,BV$DKT4 

TRANSFER  ,RTF4 

m***** ******************* 

*  * 

•  RT  TO  L(1)  ,1.(2)  ,t(3)  ♦ 

*  * 

************************* 


8TF4 

ASSIGN 

11,0 

OSS 

HACBO 

KRP4,X$KEX 

TEST  E 

BV$RTOK4,1 

ENTER 

TIKI 

ENTER 

TIK2 

ENTER 

TIK3 

SEIZE 

G3US 

ADVANCE 

XSBEX3 

ASSIGN 

44,X$BEX3 

RELEASE 

GBUS 

LEAVE 

TOK4 

SPLIT 

1,STOE1 

SPLIT 

1 , STCfi2 

SPLIT 

TERMINATE 

1,STOB3 

* 

* 

* 

*■ 


STORE  INTO  L(3)  AS  A  RESULT  OF  BEAD-THBOOGH 


STOR3  ASSIGN  11,0 

OSE  MACRO  NRP3,XSKEX 

SEND  MACRO  TIK3,TIB3,LBUS3,X$BEX3,BV$KET3 

OSS  HACBO  £BP3,Z$BEX 
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SPLIT  1,OV»3 

TRANSFER  .5,SSS31, SSS32 

************** 

*  * 

*  STORE  INTO  D31  * 

*  * 
«***«**« 


SSS31 

ASSIGN 

11,0 

SEND 

MACRO 

TIE3,TID31,1BDS3,X$BEX3,BV$E DT3 1 

OSE 

MACRO 

DRP31,X$DEX3 

LEAVE 

TID31 

'  -  - 

•  TERMINATE 

************************* 
*  * 

*  STORE  INTO  D32  * 

*  * 
************************* 


SSS32  ASSIGN 

SEND  MACRO 

USE  MACRO 

LEAVE 

TERMINATE 


11,0 

TIR3,TID32,L6US3,X$ B£X3,BVSRDT3 2 

DBP32,X$DSX3 

TID32 


*  * 

*  OVERFLOW  HANDLING  * 

*  * 
************************* 


OVH3 

TRANSFER 

.X$PCV3,NOVL3,OVL3 

OVL3 

TEST  E 

BVSRKOa, 1 

ENTER 

OCK3 

SEIZE 

LBUS3 

ADVANCE 

X$BEXM 

ASSIGN 

4+,X$BEXM 

RELEASE 

LBUS3 

SEND 

MACRO 

OOK3,OIK4,GBOS, X$BEXM,BV$KK034 

OSE 

MACRO 

KnP4,X$KEX 

SEND 

MACRO 

OIK4,OIR4,LBOS4,X$B£XH,BV$KR04 

OSE 

MACRO 

RBP4,X$B£X 

LEAVE 

OIB4 
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HOVI.3  TERMINATE 


«  COMMON  CODE  FOR  STOBE-BEHZND 


COHN  ASSIGN  11, C 

USE  MACRO  KRP1,X$KEX 

SEND  HACBO  SOKi , S1R2,GB0S, XSBEX1,BVSKKS 12 

USE  HACBO  KBP2,X$KEX 

SEND  HACBO  SIK2,S1R2,LBUS2,X$BEX1,BT$KBS2 

USE  MACRO  RBP2,X$REX 

TRANSFER  .5,SNS21,SHS22 

*  ♦ 

«  SB  MBITS  INTO  D21  * 

*  •  . 

***********  «*«4>***«*«««*« 

SVS21  ASSIGN  11,0 

SEND  MACRO  SIR2 ,SID21 ,EBUS2, X$BEX1,BV$BDS2 1 

OSS  MACRO  DRP21,XSDEX2 

SEND  HACBO  SI D2 1 ,SOK2,LBUS2,XSBEX2, BV$D KS2 

SPLIT  1,STB23 

ENTER  AOK2 

TRANSFER  ,ACK21 

4t***«*«**4e4t4'*****’»*4'««*4>* 

*  * 

♦  SB  MHITE  INTO  D22  * 

*  * 

•  *4i*«**«*4<«4'«*«***«**«*«« 

SMS22  ASSIGN  11,0 

SEND  HACBO  SIB2,SID22,LBUS2,XSB£X1,BV$RDS22 

USB  HACBO  DBP22,X$DEX2 

SEND  HACBO  SID22,S0K2,1BUS2,XSBEX2,BV$0KS2 

SPLIT  1,STB23 
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ENTER  AOK2 

TRANSFER  ,aCK21 


•  STORE 

-BEHIND  TO  1(3) 

STB  2  3 

ASSIGN 

11,0 

USE 

MACRO 

KBP2,X$KEX 

SEND 

MACRO 

SOK2,SIK3,GBOS, XSBEX2,BV$KKS23 

OSE 

MACRO 

KRP3,X$KEX 

SEND 

MACRO 

SIK3,SIR3,LB0S3,X$BEX2,BVSKRS3 

DSE 

MACRO 

EHP3,X$EEX 

TRANSFER 

.5,SWS31,SWS32 

* 

♦ 

•  SB  WHITE  INTO 

D31  ♦  . 

* 

SBS31 

ASSIGN 

11,0 

SEND 

MACRO 

SIB3,SID3 1,LBUS3,X$BEX2,BV$EDS31 

DSE 

MACRO 

DRP31,X$BEX3 

SEND 

MACRO 

SID31,S0K3,L3US3,X$BEX3,BV$DKS3 

SPLIT 

1,STB34 

ENTER 

AOK3 

TRANSFER 

,ACK32 

* 

SB  HBITE  INTO 

D32  * 

* 

^^l^m^************  ******** 

SNS32 

ASSIGN 

11,0 

SEND 

MACRO 

SIE3,S1D32,LB0S3,X$BEX2,BV$EDS32 

USE 

MACRO 

DRP32,X$DEX3 

SEND 

MACRO 

S1D32,S0K3,LBUS3,X$BEX3,  BV$DRS3 
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SPLIT 

1,STB34 

ENTER 

AOK3 

TRANSFER 

,ACX32 

STORE-BEHIND  TO  L  (4) 


STB34 

ASSIGN 

11,0 

OSE 

HACBO 

KBP3,X$KEX 

SEND 

HACRO 

SOK3,SlK4,GBas,X$BEX3,BV$KKS34 

BSE 

HACBO 

KBP4,X$KEX 

SEND 

HACRO 

SIK4,SIR4,LBUS4,X$BEX3,BV$XBS4 

OSE 

HACRO 

RRP4,X$REX 

TRANSFER 

.5,SWS41,SHS42 

*:*****  t***********^**!^** 
*  « 

*  SB  HHITE  IHTO  D41  * 

*  * 
t********** ************** 


SHS41  ASSIGN 
SEND  NACRO 
USE  HACBO 
SEND  HACBO 

TRANSFER 


11,0 

SIR4 ,S1D4 1,LDUS4, X$BEX3 ,BT$fiDS4 1 
OBP41,X$DEX4 

S1D41,A0K4,IBUS4,XSBEXH,BV$DKA4 

,ACK43 


*  * 

*  SB  BRITE  INTO  D42  * 

*  * 

*!»«**«***  :|r  ««««**  4i  *  4c  « 


SVS42  ASSIGN 
SEND  HACRO 
USE  HACBO 
SEND  HACBO 

TBANSFEB 


11,0 

SIB4,SI042,1.BUS4,X$BEX3,  BV$BDS42 
DRP42,XSD£X4 

SID42,AOK4,LBUS4,X$BBXH,BVSDKA4 

,ACK43 
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ACK  FROn  1.(4)  TO  L(3) 

ACK43  ASSIGN  11,0 

OSE  HACRO  KRP4,X$K£X 

SEND  BACRO  AOK4,AIK3,6BOS,X$BEXH,BVSKKA43 

OSE  HACRO  KfiP3,X$KEX 

SEND  HACRO  A1K3,A1R3,LBUS3,X$BEXN,BT$KRA3 

OSE  HACRO  RRP3,X$REX 

•  «**«««  ««««*«  4iiii  4. 

*  * 

*  FORWARD  THE  ACK  UP  * 

*  * 

♦♦♦••♦♦♦♦****♦♦♦*#**♦**♦* 


SEND 

HACRO 

AXR3,A0K3,I.BUS3,X$8£XH,BV$RKA3 

OSE 

HACRO 

KRP3,X$REX 

SEND 

HACRO 

A0K3,AIX2,GBUS,X$BEXH,0V$KKA32 

USE 

HACRO 

KRP2,X$KEX 

SEND 

HACRO 

AlK2,AXR2,LOaS2,XSBSXH,BV$KBA2 

OSE 

HACRO 

RBP2,XSBEX 

I.EAVE 

TEBHINATE 

AIB2 

r 

‘  ACK 

FROH  L(3} 

TO  1.(2) 

ACK32  ASSIGN  11,0 

OSE  HACRO  RRP3,X$KEX 

SEND  HACRO  A0K3,AIK2,GDUS,X$BEXH,BT$KKA32 

OSE  HACRO  KRP2,X$XEX 

SEND  HACRO  AXK2, AIR2,LBUS2, XSBEXH,BV$KR A2 
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OSS  HACBO  BBP2,XSREX 

SeKD  HACBO  A1B2,A0K2,LBUS2,X$BEXB,B7$BKA2 

TEANSFER  ,ACK21 

- - - - — - — — — - - - 

*  ■  .  * 

*  ACK  FBOB  L(2)  TO  1(1)  f 

«  * 

ACK21  ASSIG8  11,0 

USE  MACRO  KBP2,X$K£X 

SEMD  HACBO  AOK2,AIK1 ,GBOS,XSBEXH, BV$KKA21 

USE  HACBO  KBP1,X$KEX 

SPLIT  1,FS$HICHA,1 

TERM IB ATE  .  . 

*0i$0iiiti*inni0*4f**m******** 

*  ■  * 

*  ACK  HANDLED  BY  Dll  * 

♦  ♦ 

***«4i*4i««4i******«««4i4t4i«** 

AAA11  ASSIGN  11,0 

SEND  MACRO  AIK1 ,AI01 1,LBUS1 ,X$6EXH,BTSK0Al 1 

OSE  MACRO  ORP11,X$BEX 

LEAVE  A1D11 

TERMINATE 

**#**«4I«««*4I*<I>*««%««****|» 

♦  • 

*  ACK  HANDLED  BY  D12  « 

IKi^t********************** 

AAA12  ASSIGN  11,0 

SEND  MACRO  AIK1,A1012,LBaSl,X$B£XM,BT$KDA12 

OSS  MACRO  DBP12,X$REX 

LEAVE  AID12 


TERMINATE 

<»********4<«*«**«**«*****« 
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•  * 

♦  ACK  HANDLED  DY  D13  * 

*  * 

*  «  *  «  4c  «  «  *  «  *  *  «  «  « >K  «  4l  4: 4l 

AAA13  ASSIGN  11,0 

SEND  MACRO  AIK  1 ,  AID  1 3,  LBUS 1, XSBEXIl, BVSKDAI  3 

aSE  MACRO  DEP13,X$REX 

LEAVE  AID13 

tERMINATE 

«  «  «  «  «  «  «  *  K  «  4c  *  «  4c  *  «  4c «  4c  4c  14  4  4c «  « 

*  * 

•  ACK  HANDLED  BY  D14  * 

«  4c 

4c  4c  *  «  4c  4c  c4  4c «  4c «  4c  4c  4c  4c  4c  4  4c  4c «  4c  4<  *  4c  c4 

AAA14  ASSIGN  11,0 

SEND  MACRO  AIK  1 , AIDl 4, LBOS 1, X$ BEXM,By$KDAl 4 

USE  MACRO  DaP14,X$EEX 

LEAVE  AID14 

TERMINATE 

•  4i^4cc44c4ic44>4i4c4c4c4c4c4c4c4c4<4c4c4c4c44c 

4c  4c 

•  ACK  HANDLED  BY  D15  * 

4t  4c 

4c «  4>  c4  c4  4c  4c  4c  4c  4c  4c  4c  4  4c  4c  4c  4c  4c  4c  4c  4: 4c  4c  4c  4c 

AAA15  ASSIGN  11,0 

SEND  MACRO  A  IK  1 , AID  1 5, LBOS 1 , X$ BEXM, BVSKDAI  5 

OSE  MACRO  DRP15,X$EEX 

LEAVE  AID15 

TERMINATE 

4c - - - - - - - - - - 

4c 

•  SIMOLATION  CONTROL 

4i 

4c—.— - - - - - - - - - - — - — - 

GENERATE  XSTIMER 

TERMINATE  1 

START  1 

END 
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